2024-04-25T13:55:24.924 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T13:55:24.924 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} --owner scheduled_pdonnell@teuthology --verbose --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/orig.config.yaml 2024-04-25T13:55:24.968 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T13:55:25.049 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} email: pdonnell@redhat.com first_in_suite: false job_id: '7673193' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: syntax: v2 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1pqlg+w8go4L1U+8SvxXDeMrWNBQT2Q2eokT/agJHjTJ5ZqmemvfrXIl3V4jzsNgUbtcptsdy8RP6Y7HdO5ko= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1nZRCwfzW400zFnxkxoap6HYcxL2bfWFlP+6ulqjlmO1QwT0SykYcj8x3+lWYxpuWdY6lSqeR6qcthHq4Y6Ec= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_openfiletable teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T13:55:25.049 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T13:55:25.050 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T13:55:25.050 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T13:55:25.051 INFO:teuthology.task.internal:Checking packages... 2024-04-25T13:55:25.074 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T13:55:25.074 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T13:55:25.074 INFO:teuthology.packaging:ref: None 2024-04-25T13:55:25.075 INFO:teuthology.packaging:tag: None 2024-04-25T13:55:25.075 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T13:55:25.075 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:55:25.075 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T13:55:25.219 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T13:55:25.220 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T13:55:25.228 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T13:55:25.228 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T13:55:25.237 INFO:teuthology.task.internal:Saving configuration 2024-04-25T13:55:25.254 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T13:55:25.262 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T13:55:25.285 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi150.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 13:50:20.308415', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1pqlg+w8go4L1U+8SvxXDeMrWNBQT2Q2eokT/agJHjTJ5ZqmemvfrXIl3V4jzsNgUbtcptsdy8RP6Y7HdO5ko='} 2024-04-25T13:55:25.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 13:50:20.307226', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1nZRCwfzW400zFnxkxoap6HYcxL2bfWFlP+6ulqjlmO1QwT0SykYcj8x3+lWYxpuWdY6lSqeR6qcthHq4Y6Ec='} 2024-04-25T13:55:25.307 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T13:55:25.316 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-25T13:55:25.316 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-25T13:55:25.316 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T13:55:25.375 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7efc67b6af70>, signals=[15]) 2024-04-25T13:55:25.375 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T13:55:25.383 INFO:teuthology.task.internal:Opening connections... 2024-04-25T13:55:25.384 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T13:55:25.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:55:25.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T13:55:25.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:55:25.538 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T13:55:25.549 DEBUG:teuthology.orchestra.run.smithi150:> uname -m 2024-04-25T13:55:25.567 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2024-04-25T13:55:25.568 DEBUG:teuthology.orchestra.run.smithi150:> cat /etc/os-release 2024-04-25T13:55:25.625 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2024-04-25T13:55:25.625 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="9" 2024-04-25T13:55:25.625 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2024-04-25T13:55:25.625 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="9" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el9" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:LOGO="fedora-logo-icon" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T13:55:25.626 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T13:55:25.627 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2024-04-25T13:55:25.648 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2024-04-25T13:55:25.666 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2024-04-25T13:55:25.667 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2024-04-25T13:55:25.723 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2024-04-25T13:55:25.723 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="9" 2024-04-25T13:55:25.723 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="9" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el9" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:LOGO="fedora-logo-icon" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T13:55:25.724 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T13:55:25.725 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2024-04-25T13:55:25.990 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T13:55:26.000 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T13:55:26.006 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T13:55:26.007 DEBUG:teuthology.orchestra.run.smithi150:> test '!' -e /home/ubuntu/cephtest 2024-04-25T13:55:26.009 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2024-04-25T13:55:26.025 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T13:55:26.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T13:55:26.033 DEBUG:teuthology.orchestra.run.smithi150:> test -z $(ls -A /var/lib/ceph) 2024-04-25T13:55:26.065 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2024-04-25T13:55:26.091 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T13:55:26.149 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T13:55:26.163 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T13:55:26.164 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2024-04-25T13:55:26.164 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2024-04-25T13:55:26.164 DEBUG:teuthology.orchestra.run.smithi150:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T13:55:26.165 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T13:55:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:55:26.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:55:26.181 DEBUG:teuthology.orchestra.run.smithi150:> uname -r 2024-04-25T13:55:26.182 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2024-04-25T13:55:26.238 INFO:teuthology.orchestra.run.smithi187.stdout:5.14.0-437.el9.x86_64 2024-04-25T13:55:26.238 INFO:teuthology.task.kernel:Running kernel on smithi187: 5.14.0-437.el9.x86_64 2024-04-25T13:55:26.238 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2024-04-25T13:55:26.239 INFO:teuthology.orchestra.run.smithi150.stdout:5.14.0-437.el9.x86_64 2024-04-25T13:55:26.239 INFO:teuthology.task.kernel:Running kernel on smithi150: 5.14.0-437.el9.x86_64 2024-04-25T13:55:26.239 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum install -y kernel 2024-04-25T13:55:28.949 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-04-25T13:55:29.117 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.0 MB 00:01 2024-04-25T13:55:30.875 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-04-25T13:55:31.876 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-04-25T13:55:35.159 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-25T13:55:36.344 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.2 MB 00:00 2024-04-25T13:55:36.949 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-25T13:55:37.792 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-04-25T13:55:38.011 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-25T13:55:39.476 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-25T13:55:43.066 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-04-25T13:55:44.668 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 8.2 kB/s | 1.7 kB 00:00 2024-04-25T13:55:44.956 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T13:55:44.956 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T13:55:45.051 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-25T13:55:45.052 INFO:teuthology.orchestra.run.smithi187.stdout:Nothing to do. 2024-04-25T13:55:45.053 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-25T13:55:45.159 DEBUG:teuthology.orchestra.run.smithi187:> echo no | sudo yum reinstall kernel || true 2024-04-25T13:55:45.657 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:02 ago on Thu 25 Apr 2024 01:55:43 PM UTC. 2024-04-25T13:55:45.776 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalling: 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 255 k 2024-04-25T13:55:45.785 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 0 2024-04-25T13:55:45.786 INFO:teuthology.orchestra.run.smithi187.stderr:Operation aborted. 2024-04-25T13:55:45.844 INFO:teuthology.orchestra.run.smithi187.stdout:Is this ok [y/N]: 2024-04-25T13:55:45.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum reinstall -y kernel || true 2024-04-25T13:55:46.340 INFO:teuthology.orchestra.run.smithi187.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:55:43 PM UTC. 2024-04-25T13:55:46.456 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-25T13:55:46.456 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:46.456 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalling: 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 255 k 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 0 2024-04-25T13:55:46.457 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-25T13:55:46.541 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T13:55:46.541 INFO:teuthology.orchestra.run.smithi150.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T13:55:46.634 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-25T13:55:46.635 INFO:teuthology.orchestra.run.smithi150.stdout:Nothing to do. 2024-04-25T13:55:46.635 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-25T13:55:46.743 DEBUG:teuthology.orchestra.run.smithi150:> echo no | sudo yum reinstall kernel || true 2024-04-25T13:55:47.210 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-437.el9.x86_64.rpm 608 kB/s | 255 kB 00:00 2024-04-25T13:55:47.210 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:55:47.210 INFO:teuthology.orchestra.run.smithi187.stdout:Total 338 kB/s | 255 kB 00:00 2024-04-25T13:55:47.226 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-25T13:55:47.284 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:55:44 PM UTC. 2024-04-25T13:55:47.402 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-25T13:55:47.405 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-25T13:55:47.405 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-25T13:55:47.410 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-25T13:55:47.410 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalling: 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:47.412 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:47.413 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 255 k 2024-04-25T13:55:47.413 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 0 2024-04-25T13:55:47.413 INFO:teuthology.orchestra.run.smithi150.stderr:Operation aborted. 2024-04-25T13:55:47.469 INFO:teuthology.orchestra.run.smithi150.stdout:Is this ok [y/N]: 2024-04-25T13:55:47.470 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum reinstall -y kernel || true 2024-04-25T13:55:47.572 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-25T13:55:47.644 INFO:teuthology.orchestra.run.smithi187.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:55:47.645 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:47.971 INFO:teuthology.orchestra.run.smithi150.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 01:55:44 PM UTC. 2024-04-25T13:55:48.086 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-25T13:55:48.087 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:48.087 INFO:teuthology.orchestra.run.smithi150.stdout: Package Architecture Version Repository Size 2024-04-25T13:55:48.087 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:48.087 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalling: 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:================================================================================ 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 255 k 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Installed size: 0 2024-04-25T13:55:48.088 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-25T13:55:48.358 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:48.358 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:55:48.537 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-437.el9.x86_64.rpm 996 kB/s | 255 kB 00:00 2024-04-25T13:55:48.537 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:55:48.537 INFO:teuthology.orchestra.run.smithi150.stdout:Total 566 kB/s | 255 kB 00:00 2024-04-25T13:55:48.560 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-25T13:55:48.730 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-25T13:55:48.730 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-25T13:55:48.735 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-25T13:55:48.736 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-25T13:55:48.871 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-25T13:55:48.930 INFO:teuthology.orchestra.run.smithi150.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:55:48.930 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout:Reinstalled: 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:55:49.040 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-25T13:55:49.213 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T13:55:49.260 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T13:55:49.261 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T13:55:49.261 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T13:55:49.261 DEBUG:teuthology.parallel:result is None 2024-04-25T13:55:49.672 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:49.673 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout:Reinstalled: 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:55:50.383 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-25T13:55:50.566 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T13:55:50.612 INFO:teuthology.orchestra.run.smithi150.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T13:55:50.612 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T13:55:50.612 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T13:55:50.612 DEBUG:teuthology.parallel:result is None 2024-04-25T13:55:50.613 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T13:55:50.621 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T13:55:50.621 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T13:55:50.623 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T13:55:50.641 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T13:55:50.648 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T13:55:50.658 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T13:55:50.658 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T13:55:50.679 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T13:55:50.715 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T13:55:50.721 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T13:55:50.722 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T13:55:50.744 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T13:55:50.778 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:55:50.790 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:55:50.805 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:55:50.821 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T13:55:50.822 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T13:55:50.830 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T13:55:50.830 DEBUG:teuthology.orchestra.run.smithi150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T13:55:50.834 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T13:55:50.859 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T13:55:50.867 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T13:55:50.868 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T13:55:50.898 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T13:55:50.915 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:55:50.994 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:55:51.041 DEBUG:teuthology.orchestra.run.smithi150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:55:51.122 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:55:51.162 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T13:55:51.162 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T13:55:51.224 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:55:51.265 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T13:55:51.320 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:55:51.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T13:55:51.443 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T13:55:51.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T13:55:51.508 DEBUG:teuthology.orchestra.run.smithi150:> sudo service rsyslog restart 2024-04-25T13:55:51.510 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2024-04-25T13:55:51.545 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T13:55:51.589 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T13:55:52.043 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T13:55:52.051 INFO:teuthology.task.internal:Starting timer... 2024-04-25T13:55:52.051 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T13:55:52.061 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T13:55:52.069 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-25T13:55:52.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo service auditd rotate 2024-04-25T13:55:52.153 INFO:teuthology.orchestra.run.smithi150.stdout:Rotating logs: 2024-04-25T13:55:52.155 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2024-04-25T13:55:52.193 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2024-04-25T13:55:52.195 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T13:55:52.195 DEBUG:teuthology.orchestra.run.smithi150:> /usr/sbin/getenforce 2024-04-25T13:55:52.217 INFO:teuthology.orchestra.run.smithi150.stdout:Permissive 2024-04-25T13:55:52.218 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2024-04-25T13:55:52.257 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2024-04-25T13:55:52.258 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi150.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2024-04-25T13:55:52.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T13:55:52.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:55:52.282 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T13:55:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T13:55:52.322 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T13:55:52.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo /usr/sbin/setenforce permissive 2024-04-25T13:55:52.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2024-04-25T13:55:52.386 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T13:55:52.401 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T13:55:52.418 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T13:55:52.420 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi150.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T13:59:21.227 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi150.front.sepia.ceph.com'), Remote(name='ubuntu@smithi187.front.sepia.ceph.com')] 2024-04-25T13:59:21.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T13:59:21.230 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:59:21.311 DEBUG:teuthology.orchestra.run.smithi150:> true 2024-04-25T13:59:21.385 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi150.front.sepia.ceph.com' 2024-04-25T13:59:21.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-25T13:59:21.386 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T13:59:21.462 DEBUG:teuthology.orchestra.run.smithi187:> true 2024-04-25T13:59:21.494 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2024-04-25T13:59:21.495 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T13:59:21.508 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T13:59:21.508 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T13:59:21.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T13:59:21.511 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T13:59:21.512 DEBUG:teuthology.orchestra.run.smithi187:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T13:59:21.536 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T13:59:21.550 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T13:59:21.562 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T13:59:21.572 INFO:teuthology.orchestra.run.smithi150.stderr:sudo: ntpd: command not found 2024-04-25T13:59:21.575 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T13:59:21.582 INFO:teuthology.orchestra.run.smithi150.stdout:506 Cannot talk to daemon 2024-04-25T13:59:21.595 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T13:59:21.598 INFO:teuthology.orchestra.run.smithi187.stderr:sudo: ntpd: command not found 2024-04-25T13:59:21.607 INFO:teuthology.orchestra.run.smithi150.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T13:59:21.608 INFO:teuthology.orchestra.run.smithi187.stdout:506 Cannot talk to daemon 2024-04-25T13:59:21.621 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T13:59:21.633 INFO:teuthology.orchestra.run.smithi187.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T13:59:21.659 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-04-25T13:59:21.662 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T13:59:21.662 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-04-25T13:59:21.662 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.662 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.663 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.663 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.689 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-25T13:59:21.692 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T13:59:21.693 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-25T13:59:21.693 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.693 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.693 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.693 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T13:59:21.693 INFO:teuthology.run_tasks:Running task install... 2024-04-25T13:59:21.703 DEBUG:teuthology.task.install:project ceph 2024-04-25T13:59:21.703 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T13:59:21.703 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T13:59:21.703 INFO:teuthology.task.install:Using flavor: default 2024-04-25T13:59:21.710 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T13:59:21.710 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T13:59:21.711 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T13:59:21.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:59:21.713 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T13:59:21.714 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T13:59:21.858 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T13:59:21.859 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T13:59:21.929 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T13:59:21.929 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T13:59:22.033 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T13:59:22.033 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T13:59:22.033 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T13:59:22.059 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T13:59:22.060 DEBUG:teuthology.orchestra.run.smithi150:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T13:59:22.112 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T13:59:22.112 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T13:59:22.112 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T13:59:22.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T13:59:22.137 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T13:59:22.137 DEBUG:teuthology.orchestra.run.smithi187:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T13:59:22.198 DEBUG:teuthology.orchestra.run.smithi150:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T13:59:22.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T13:59:22.260 INFO:teuthology.orchestra.run.smithi150.stdout:check_obsoletes = 1 2024-04-25T13:59:22.262 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum clean all 2024-04-25T13:59:22.274 DEBUG:teuthology.orchestra.run.smithi187:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T13:59:22.337 INFO:teuthology.orchestra.run.smithi187.stdout:check_obsoletes = 1 2024-04-25T13:59:22.338 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum clean all 2024-04-25T13:59:22.485 INFO:teuthology.orchestra.run.smithi150.stdout:48 files removed 2024-04-25T13:59:22.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T13:59:22.562 INFO:teuthology.orchestra.run.smithi187.stdout:48 files removed 2024-04-25T13:59:22.590 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T13:59:23.429 INFO:teuthology.orchestra.run.smithi187.stdout:ceph packages for x86_64 144 kB/s | 84 kB 00:00 2024-04-25T13:59:23.509 INFO:teuthology.orchestra.run.smithi150.stdout:ceph packages for x86_64 114 kB/s | 84 kB 00:00 2024-04-25T13:59:24.246 INFO:teuthology.orchestra.run.smithi187.stdout:ceph noarch packages 15 kB/s | 12 kB 00:00 2024-04-25T13:59:24.263 INFO:teuthology.orchestra.run.smithi150.stdout:ceph noarch packages 16 kB/s | 12 kB 00:00 2024-04-25T13:59:24.889 INFO:teuthology.orchestra.run.smithi187.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-25T13:59:24.956 INFO:teuthology.orchestra.run.smithi150.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-04-25T13:59:25.516 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-25T13:59:25.766 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-04-25T13:59:27.817 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-25T13:59:29.459 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - AppStream 7.2 MB/s | 19 MB 00:02 2024-04-25T13:59:32.118 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-04-25T13:59:33.918 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - CRB 6.8 MB/s | 6.2 MB 00:00 2024-04-25T13:59:34.800 INFO:teuthology.orchestra.run.smithi150.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-04-25T13:59:35.549 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-25T13:59:36.629 INFO:teuthology.orchestra.run.smithi150.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-25T13:59:41.362 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 3.7 MB/s | 21 MB 00:05 2024-04-25T13:59:41.777 INFO:teuthology.orchestra.run.smithi150.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-25T13:59:43.903 INFO:teuthology.orchestra.run.smithi150.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:59:43.904 INFO:teuthology.orchestra.run.smithi150.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:59:43.914 INFO:teuthology.orchestra.run.smithi150.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T13:59:43.915 INFO:teuthology.orchestra.run.smithi150.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T13:59:43.916 INFO:teuthology.orchestra.run.smithi150.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T13:59:43.917 INFO:teuthology.orchestra.run.smithi150.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T13:59:43.918 INFO:teuthology.orchestra.run.smithi150.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T13:59:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T13:59:43.921 INFO:teuthology.orchestra.run.smithi150.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T13:59:43.981 INFO:teuthology.orchestra.run.smithi150.stdout:Dependencies resolved. 2024-04-25T13:59:43.990 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-25T13:59:43.990 INFO:teuthology.orchestra.run.smithi150.stdout: Package Arch Version Repository Size 2024-04-25T13:59:43.990 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-25T13:59:43.990 INFO:teuthology.orchestra.run.smithi150.stdout:Installing: 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T13:59:43.991 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T13:59:43.992 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrading: 2024-04-25T13:59:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout:Installing dependencies: 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T13:59:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T13:59:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T13:59:43.996 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T13:59:43.997 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T13:59:43.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T13:59:43.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T13:59:43.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T13:59:43.998 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T13:59:43.999 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T13:59:44.000 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T13:59:44.001 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T13:59:44.002 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout:Installing weak dependencies: 2024-04-25T13:59:44.003 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction Summary 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout:======================================================================================= 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout:Install 148 Packages 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout:Upgrade 4 Packages 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T13:59:44.004 INFO:teuthology.orchestra.run.smithi150.stdout:Total download size: 214 M 2024-04-25T13:59:44.005 INFO:teuthology.orchestra.run.smithi150.stdout:Downloading Packages: 2024-04-25T13:59:44.587 INFO:teuthology.orchestra.run.smithi150.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T13:59:44.762 INFO:teuthology.orchestra.run.smithi150.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 4.8 MB/s | 863 kB 00:00 2024-04-25T13:59:44.812 INFO:teuthology.orchestra.run.smithi150.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.9 MB/s | 147 kB 00:00 2024-04-25T13:59:45.079 INFO:teuthology.orchestra.run.smithi150.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 8.5 MB/s | 2.3 MB 00:00 2024-04-25T13:59:45.271 INFO:teuthology.orchestra.run.smithi150.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 7.9 MB/s | 1.5 MB 00:00 2024-04-25T13:59:45.380 INFO:teuthology.orchestra.run.smithi150.stdout:(6/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 5.6 MB/s | 5.4 MB 00:00 2024-04-25T13:59:45.772 INFO:teuthology.orchestra.run.smithi150.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 9.7 MB/s | 4.9 MB 00:00 2024-04-25T13:59:46.433 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-04-25T13:59:47.022 INFO:teuthology.orchestra.run.smithi150.stdout:(8/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 11 MB/s | 18 MB 00:01 2024-04-25T13:59:47.180 INFO:teuthology.orchestra.run.smithi150.stdout:(9/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 7.2 MB/s | 10 MB 00:01 2024-04-25T13:59:47.206 INFO:teuthology.orchestra.run.smithi150.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 137 kB/s | 25 kB 00:00 2024-04-25T13:59:47.297 INFO:teuthology.orchestra.run.smithi150.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.4 MB/s | 224 kB 00:00 2024-04-25T13:59:47.398 INFO:teuthology.orchestra.run.smithi150.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 270 kB/s | 27 kB 00:00 2024-04-25T13:59:47.498 INFO:teuthology.orchestra.run.smithi150.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 317 kB/s | 32 kB 00:00 2024-04-25T13:59:47.615 INFO:teuthology.orchestra.run.smithi150.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 6.3 MB/s | 756 kB 00:00 2024-04-25T13:59:47.723 INFO:teuthology.orchestra.run.smithi150.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.4 MB/s | 158 kB 00:00 2024-04-25T13:59:47.907 INFO:teuthology.orchestra.run.smithi150.stdout:(16/152): ceph-common-19.0.0-3291.g994e2f20.el9 5.6 MB/s | 20 MB 00:03 2024-04-25T13:59:47.932 INFO:teuthology.orchestra.run.smithi150.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 608 kB/s | 127 kB 00:00 2024-04-25T13:59:47.991 INFO:teuthology.orchestra.run.smithi150.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 5.8 MB/s | 497 kB 00:00 2024-04-25T13:59:48.083 INFO:teuthology.orchestra.run.smithi150.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 493 kB/s | 45 kB 00:00 2024-04-25T13:59:48.183 INFO:teuthology.orchestra.run.smithi150.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.3 MB/s | 135 kB 00:00 2024-04-25T13:59:48.284 INFO:teuthology.orchestra.run.smithi150.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.6 MB/s | 161 kB 00:00 2024-04-25T13:59:48.392 INFO:teuthology.orchestra.run.smithi150.stdout:(22/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 11 MB/s | 5.2 MB 00:00 2024-04-25T13:59:48.417 INFO:teuthology.orchestra.run.smithi150.stdout:(23/152): python3-rados-19.0.0-3291.g994e2f20.e 2.3 MB/s | 321 kB 00:00 2024-04-25T13:59:48.476 INFO:teuthology.orchestra.run.smithi150.stdout:(24/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.5 MB/s | 298 kB 00:00 2024-04-25T13:59:48.502 INFO:teuthology.orchestra.run.smithi150.stdout:(25/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.2 MB/s | 100 kB 00:00 2024-04-25T13:59:48.544 INFO:teuthology.orchestra.run.smithi150.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T13:59:48.567 INFO:teuthology.orchestra.run.smithi187.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:59:48.568 INFO:teuthology.orchestra.run.smithi187.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T13:59:48.578 INFO:teuthology.orchestra.run.smithi187.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T13:59:48.579 INFO:teuthology.orchestra.run.smithi187.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T13:59:48.580 INFO:teuthology.orchestra.run.smithi187.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T13:59:48.581 INFO:teuthology.orchestra.run.smithi187.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T13:59:48.582 INFO:teuthology.orchestra.run.smithi187.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T13:59:48.585 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T13:59:48.585 INFO:teuthology.orchestra.run.smithi187.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T13:59:48.619 INFO:teuthology.orchestra.run.smithi150.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.2 MB/s | 169 kB 00:00 2024-04-25T13:59:48.645 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2024-04-25T13:59:48.654 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: Package Arch Version Repository Size 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T13:59:48.655 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T13:59:48.656 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrading: 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T13:59:48.657 INFO:teuthology.orchestra.run.smithi187.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T13:59:48.658 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T13:59:48.659 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T13:59:48.660 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T13:59:48.661 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T13:59:48.662 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T13:59:48.663 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T13:59:48.664 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T13:59:48.665 INFO:teuthology.orchestra.run.smithi187.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout:Installing weak dependencies: 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout:======================================================================================= 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout:Install 148 Packages 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout:Upgrade 4 Packages 2024-04-25T13:59:48.666 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T13:59:48.667 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 214 M 2024-04-25T13:59:48.667 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2024-04-25T13:59:48.703 INFO:teuthology.orchestra.run.smithi150.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 409 kB/s | 34 kB 00:00 2024-04-25T13:59:48.786 INFO:teuthology.orchestra.run.smithi150.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.7 MB/s | 142 kB 00:00 2024-04-25T13:59:48.895 INFO:teuthology.orchestra.run.smithi150.stdout:(30/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 7.7 MB/s | 3.0 MB 00:00 2024-04-25T13:59:49.145 INFO:teuthology.orchestra.run.smithi150.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 9.9 MB/s | 3.5 MB 00:00 2024-04-25T13:59:49.245 INFO:teuthology.orchestra.run.smithi150.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T13:59:49.329 INFO:teuthology.orchestra.run.smithi150.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 591 kB/s | 49 kB 00:00 2024-04-25T13:59:49.421 INFO:teuthology.orchestra.run.smithi150.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 179 kB/s | 16 kB 00:00 2024-04-25T13:59:49.505 INFO:teuthology.orchestra.run.smithi150.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 3.2 MB/s | 273 kB 00:00 2024-04-25T13:59:49.605 INFO:teuthology.orchestra.run.smithi150.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 7.4 MB/s | 762 kB 00:00 2024-04-25T13:59:49.680 INFO:teuthology.orchestra.run.smithi150.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 243 kB/s | 18 kB 00:00 2024-04-25T13:59:49.839 INFO:teuthology.orchestra.run.smithi150.stdout:(38/152): ceph-mgr-diskprediction-local-19.0.0- 7.8 MB/s | 7.4 MB 00:00 2024-04-25T13:59:49.864 INFO:teuthology.orchestra.run.smithi150.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 438 kB/s | 80 kB 00:00 2024-04-25T13:59:49.892 INFO:teuthology.orchestra.run.smithi187.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-04-25T13:59:49.931 INFO:teuthology.orchestra.run.smithi150.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 8.5 MB/s | 803 kB 00:00 2024-04-25T13:59:49.956 INFO:teuthology.orchestra.run.smithi150.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-25T13:59:49.982 INFO:teuthology.orchestra.run.smithi150.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T13:59:50.024 INFO:teuthology.orchestra.run.smithi150.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 18 MB/s | 1.2 MB 00:00 2024-04-25T13:59:50.049 INFO:teuthology.orchestra.run.smithi150.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-04-25T13:59:50.074 INFO:teuthology.orchestra.run.smithi150.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T13:59:50.099 INFO:teuthology.orchestra.run.smithi150.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T13:59:50.125 INFO:teuthology.orchestra.run.smithi150.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T13:59:50.142 INFO:teuthology.orchestra.run.smithi187.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 3.4 MB/s | 863 kB 00:00 2024-04-25T13:59:50.150 INFO:teuthology.orchestra.run.smithi150.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 14 MB/s | 689 kB 00:00 2024-04-25T13:59:50.192 INFO:teuthology.orchestra.run.smithi150.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T13:59:50.209 INFO:teuthology.orchestra.run.smithi150.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T13:59:50.218 INFO:teuthology.orchestra.run.smithi187.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.9 MB/s | 147 kB 00:00 2024-04-25T13:59:50.234 INFO:teuthology.orchestra.run.smithi150.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 514 kB/s | 21 kB 00:00 2024-04-25T13:59:50.260 INFO:teuthology.orchestra.run.smithi150.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T13:59:50.285 INFO:teuthology.orchestra.run.smithi150.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 660 kB/s | 33 kB 00:00 2024-04-25T13:59:50.402 INFO:teuthology.orchestra.run.smithi150.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-04-25T13:59:50.427 INFO:teuthology.orchestra.run.smithi150.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 18 kB 00:00 2024-04-25T13:59:50.477 INFO:teuthology.orchestra.run.smithi150.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-25T13:59:50.503 INFO:teuthology.orchestra.run.smithi150.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-25T13:59:50.528 INFO:teuthology.orchestra.run.smithi150.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 898 kB/s | 45 kB 00:00 2024-04-25T13:59:50.568 INFO:teuthology.orchestra.run.smithi187.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 6.5 MB/s | 2.3 MB 00:00 2024-04-25T13:59:50.570 INFO:teuthology.orchestra.run.smithi150.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-25T13:59:50.604 INFO:teuthology.orchestra.run.smithi150.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-04-25T13:59:50.629 INFO:teuthology.orchestra.run.smithi150.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 442 kB/s | 25 kB 00:00 2024-04-25T13:59:50.654 INFO:teuthology.orchestra.run.smithi150.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-25T13:59:50.679 INFO:teuthology.orchestra.run.smithi150.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T13:59:50.705 INFO:teuthology.orchestra.run.smithi150.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T13:59:50.730 INFO:teuthology.orchestra.run.smithi150.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-04-25T13:59:50.893 INFO:teuthology.orchestra.run.smithi187.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 4.6 MB/s | 1.5 MB 00:00 2024-04-25T13:59:50.905 INFO:teuthology.orchestra.run.smithi150.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-04-25T13:59:51.106 INFO:teuthology.orchestra.run.smithi150.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-25T13:59:51.139 INFO:teuthology.orchestra.run.smithi150.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 6.9 MB/s | 1.6 MB 00:00 2024-04-25T13:59:51.189 INFO:teuthology.orchestra.run.smithi150.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T13:59:51.214 INFO:teuthology.orchestra.run.smithi150.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-25T13:59:51.457 INFO:teuthology.orchestra.run.smithi150.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-04-25T13:59:51.490 INFO:teuthology.orchestra.run.smithi150.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-04-25T13:59:51.516 INFO:teuthology.orchestra.run.smithi150.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-25T13:59:51.608 INFO:teuthology.orchestra.run.smithi150.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.6 MB/s | 244 kB 00:00 2024-04-25T13:59:51.610 INFO:teuthology.orchestra.run.smithi187.stdout:(6/152): ceph-common-19.0.0-3291.g994e2f20.el9. 10 MB/s | 20 MB 00:01 2024-04-25T13:59:51.677 INFO:teuthology.orchestra.run.smithi187.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 6.2 MB/s | 4.9 MB 00:00 2024-04-25T13:59:51.791 INFO:teuthology.orchestra.run.smithi150.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-04-25T13:59:51.816 INFO:teuthology.orchestra.run.smithi150.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 248 kB 00:00 2024-04-25T13:59:51.850 INFO:teuthology.orchestra.run.smithi150.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 441 kB/s | 26 kB 00:00 2024-04-25T13:59:51.875 INFO:teuthology.orchestra.run.smithi150.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 327 kB/s | 19 kB 00:00 2024-04-25T13:59:51.909 INFO:teuthology.orchestra.run.smithi150.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-04-25T13:59:51.926 INFO:teuthology.orchestra.run.smithi150.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 685 kB/s | 35 kB 00:00 2024-04-25T13:59:52.385 INFO:teuthology.orchestra.run.smithi150.stdout:(81/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 8.3 MB/s | 43 MB 00:05 2024-04-25T13:59:52.418 INFO:teuthology.orchestra.run.smithi150.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 509 kB/s | 250 kB 00:00 2024-04-25T13:59:52.494 INFO:teuthology.orchestra.run.smithi150.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.7 MB/s | 5.1 MB 00:00 2024-04-25T13:59:52.519 INFO:teuthology.orchestra.run.smithi150.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-25T13:59:52.544 INFO:teuthology.orchestra.run.smithi150.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 625 kB/s | 77 kB 00:00 2024-04-25T13:59:52.569 INFO:teuthology.orchestra.run.smithi150.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 568 kB/s | 42 kB 00:00 2024-04-25T13:59:52.678 INFO:teuthology.orchestra.run.smithi150.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-04-25T13:59:52.728 INFO:teuthology.orchestra.run.smithi150.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 865 kB/s | 159 kB 00:00 2024-04-25T13:59:52.753 INFO:teuthology.orchestra.run.smithi150.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.5 MB/s | 279 kB 00:00 2024-04-25T13:59:52.795 INFO:teuthology.orchestra.run.smithi150.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 808 kB/s | 54 kB 00:00 2024-04-25T13:59:52.820 INFO:teuthology.orchestra.run.smithi150.stdout:(91/152): python3-pytz-2021.1-5.el9.noarch.rpm 556 kB/s | 51 kB 00:00 2024-04-25T13:59:53.136 INFO:teuthology.orchestra.run.smithi187.stdout:(8/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x86 12 MB/s | 18 MB 00:01 2024-04-25T13:59:53.219 INFO:teuthology.orchestra.run.smithi187.stdout:(9/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 1.5 MB/s | 5.4 MB 00:03 2024-04-25T13:59:53.244 INFO:teuthology.orchestra.run.smithi187.stdout:(10/152): ceph-selinux-19.0.0-3291.g994e2f20.el 232 kB/s | 25 kB 00:00 2024-04-25T13:59:53.362 INFO:teuthology.orchestra.run.smithi187.stdout:(11/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 6.0 MB/s | 10 MB 00:01 2024-04-25T13:59:53.387 INFO:teuthology.orchestra.run.smithi187.stdout:(12/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 1.5 MB/s | 224 kB 00:00 2024-04-25T13:59:53.396 INFO:teuthology.orchestra.run.smithi150.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 69 kB/s | 42 kB 00:00 2024-04-25T13:59:53.412 INFO:teuthology.orchestra.run.smithi187.stdout:(13/152): cephfs-shell-19.0.0-3291.g994e2f20.el 540 kB/s | 27 kB 00:00 2024-04-25T13:59:53.421 INFO:teuthology.orchestra.run.smithi150.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 74 kB/s | 44 kB 00:00 2024-04-25T13:59:53.437 INFO:teuthology.orchestra.run.smithi187.stdout:(14/152): libcephfs-devel-19.0.0-3291.g994e2f20 632 kB/s | 32 kB 00:00 2024-04-25T13:59:53.496 INFO:teuthology.orchestra.run.smithi187.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.7 MB/s | 158 kB 00:00 2024-04-25T13:59:53.529 INFO:teuthology.orchestra.run.smithi187.stdout:(16/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 6.3 MB/s | 756 kB 00:00 2024-04-25T13:59:53.555 INFO:teuthology.orchestra.run.smithi187.stdout:(17/152): librados-devel-19.0.0-3291.g994e2f20. 2.1 MB/s | 127 kB 00:00 2024-04-25T13:59:53.597 INFO:teuthology.orchestra.run.smithi187.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 7.3 MB/s | 497 kB 00:00 2024-04-25T13:59:53.655 INFO:teuthology.orchestra.run.smithi187.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 776 kB/s | 45 kB 00:00 2024-04-25T13:59:53.714 INFO:teuthology.orchestra.run.smithi187.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.3 MB/s | 135 kB 00:00 2024-04-25T13:59:53.773 INFO:teuthology.orchestra.run.smithi187.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.7 MB/s | 161 kB 00:00 2024-04-25T13:59:53.839 INFO:teuthology.orchestra.run.smithi187.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 4.7 MB/s | 321 kB 00:00 2024-04-25T13:59:53.940 INFO:teuthology.orchestra.run.smithi187.stdout:(23/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 13 MB/s | 5.2 MB 00:00 2024-04-25T13:59:53.963 INFO:teuthology.orchestra.run.smithi150.stdout:(94/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-04-25T13:59:53.965 INFO:teuthology.orchestra.run.smithi187.stdout:(24/152): python3-rbd-19.0.0-3291.g994e2f20.el9 2.3 MB/s | 298 kB 00:00 2024-04-25T13:59:53.988 INFO:teuthology.orchestra.run.smithi150.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 373 kB/s | 221 kB 00:00 2024-04-25T13:59:53.990 INFO:teuthology.orchestra.run.smithi187.stdout:(25/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.9 MB/s | 100 kB 00:00 2024-04-25T13:59:54.013 INFO:teuthology.orchestra.run.smithi150.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 61 kB/s | 36 kB 00:00 2024-04-25T13:59:54.015 INFO:teuthology.orchestra.run.smithi187.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.7 MB/s | 86 kB 00:00 2024-04-25T13:59:54.047 INFO:teuthology.orchestra.run.smithi150.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 573 kB/s | 47 kB 00:00 2024-04-25T13:59:54.072 INFO:teuthology.orchestra.run.smithi150.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 995 kB/s | 58 kB 00:00 2024-04-25T13:59:54.074 INFO:teuthology.orchestra.run.smithi187.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.8 MB/s | 169 kB 00:00 2024-04-25T13:59:54.097 INFO:teuthology.orchestra.run.smithi150.stdout:(99/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 305 kB 00:00 2024-04-25T13:59:54.122 INFO:teuthology.orchestra.run.smithi150.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T13:59:54.133 INFO:teuthology.orchestra.run.smithi187.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 588 kB/s | 34 kB 00:00 2024-04-25T13:59:54.148 INFO:teuthology.orchestra.run.smithi150.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 852 kB/s | 64 kB 00:00 2024-04-25T13:59:54.173 INFO:teuthology.orchestra.run.smithi150.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-04-25T13:59:54.198 INFO:teuthology.orchestra.run.smithi150.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 595 kB/s | 45 kB 00:00 2024-04-25T13:59:54.216 INFO:teuthology.orchestra.run.smithi187.stdout:(29/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 13 MB/s | 3.0 MB 00:00 2024-04-25T13:59:54.242 INFO:teuthology.orchestra.run.smithi187.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.3 MB/s | 142 kB 00:00 2024-04-25T13:59:54.282 INFO:teuthology.orchestra.run.smithi150.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-04-25T13:59:54.592 INFO:teuthology.orchestra.run.smithi187.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 9.4 MB/s | 3.5 MB 00:00 2024-04-25T13:59:54.666 INFO:teuthology.orchestra.run.smithi150.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 59 kB/s | 19 kB 00:00 2024-04-25T13:59:54.668 INFO:teuthology.orchestra.run.smithi187.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 3.3 MB/s | 249 kB 00:00 2024-04-25T13:59:54.768 INFO:teuthology.orchestra.run.smithi187.stdout:(33/152): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-04-25T13:59:54.802 INFO:teuthology.orchestra.run.smithi187.stdout:(34/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 367 kB/s | 49 kB 00:00 2024-04-25T13:59:54.816 INFO:teuthology.orchestra.run.smithi150.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 501 kB/s | 309 kB 00:00 2024-04-25T13:59:54.827 INFO:teuthology.orchestra.run.smithi187.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3291.g9 282 kB/s | 16 kB 00:00 2024-04-25T13:59:54.852 INFO:teuthology.orchestra.run.smithi187.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 5.4 MB/s | 273 kB 00:00 2024-04-25T13:59:54.874 INFO:teuthology.orchestra.run.smithi150.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 782 kB/s | 548 kB 00:00 2024-04-25T13:59:54.894 INFO:teuthology.orchestra.run.smithi187.stdout:(37/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 11 MB/s | 762 kB 00:00 2024-04-25T13:59:54.908 INFO:teuthology.orchestra.run.smithi150.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-25T13:59:54.919 INFO:teuthology.orchestra.run.smithi187.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 271 kB/s | 18 kB 00:00 2024-04-25T13:59:54.958 INFO:teuthology.orchestra.run.smithi150.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-04-25T13:59:55.000 INFO:teuthology.orchestra.run.smithi150.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-04-25T13:59:55.059 INFO:teuthology.orchestra.run.smithi150.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-04-25T13:59:55.087 INFO:teuthology.orchestra.run.smithi187.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 418 kB/s | 80 kB 00:00 2024-04-25T13:59:55.194 INFO:teuthology.orchestra.run.smithi150.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.2 MB/s | 838 kB 00:00 2024-04-25T13:59:55.219 INFO:teuthology.orchestra.run.smithi150.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.3 MB/s | 548 kB 00:00 2024-04-25T13:59:55.220 INFO:teuthology.orchestra.run.smithi187.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-04-25T13:59:55.245 INFO:teuthology.orchestra.run.smithi187.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-04-25T13:59:55.287 INFO:teuthology.orchestra.run.smithi187.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.7 MB/s | 253 kB 00:00 2024-04-25T13:59:55.295 INFO:teuthology.orchestra.run.smithi150.stdout:(114/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 4.4 MB 00:00 2024-04-25T13:59:55.312 INFO:teuthology.orchestra.run.smithi150.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 371 kB/s | 43 kB 00:00 2024-04-25T13:59:55.329 INFO:teuthology.orchestra.run.smithi150.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-04-25T13:59:55.346 INFO:teuthology.orchestra.run.smithi187.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-04-25T13:59:55.362 INFO:teuthology.orchestra.run.smithi150.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-25T13:59:55.371 INFO:teuthology.orchestra.run.smithi187.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-04-25T13:59:55.396 INFO:teuthology.orchestra.run.smithi187.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T13:59:55.412 INFO:teuthology.orchestra.run.smithi150.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T13:59:55.422 INFO:teuthology.orchestra.run.smithi187.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-25T13:59:55.446 INFO:teuthology.orchestra.run.smithi150.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-25T13:59:55.447 INFO:teuthology.orchestra.run.smithi187.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T13:59:55.472 INFO:teuthology.orchestra.run.smithi187.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 13 MB/s | 689 kB 00:00 2024-04-25T13:59:55.480 INFO:teuthology.orchestra.run.smithi150.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-25T13:59:55.505 INFO:teuthology.orchestra.run.smithi150.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-25T13:59:55.514 INFO:teuthology.orchestra.run.smithi187.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T13:59:55.530 INFO:teuthology.orchestra.run.smithi150.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-25T13:59:55.539 INFO:teuthology.orchestra.run.smithi187.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T13:59:55.565 INFO:teuthology.orchestra.run.smithi187.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 430 kB/s | 21 kB 00:00 2024-04-25T13:59:55.589 INFO:teuthology.orchestra.run.smithi150.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-25T13:59:55.590 INFO:teuthology.orchestra.run.smithi187.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T13:59:55.607 INFO:teuthology.orchestra.run.smithi187.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 791 kB/s | 33 kB 00:00 2024-04-25T13:59:55.614 INFO:teuthology.orchestra.run.smithi150.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-25T13:59:55.639 INFO:teuthology.orchestra.run.smithi150.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-04-25T13:59:55.673 INFO:teuthology.orchestra.run.smithi150.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-25T13:59:55.691 INFO:teuthology.orchestra.run.smithi187.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-25T13:59:55.698 INFO:teuthology.orchestra.run.smithi150.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-04-25T13:59:55.708 INFO:teuthology.orchestra.run.smithi187.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 184 kB/s | 18 kB 00:00 2024-04-25T13:59:55.724 INFO:teuthology.orchestra.run.smithi150.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 231 kB/s | 19 kB 00:00 2024-04-25T13:59:55.724 INFO:teuthology.orchestra.run.smithi187.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-25T13:59:55.749 INFO:teuthology.orchestra.run.smithi150.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-25T13:59:55.750 INFO:teuthology.orchestra.run.smithi187.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 160 kB 00:00 2024-04-25T13:59:55.775 INFO:teuthology.orchestra.run.smithi187.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 908 kB/s | 45 kB 00:00 2024-04-25T13:59:55.799 INFO:teuthology.orchestra.run.smithi150.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-25T13:59:55.808 INFO:teuthology.orchestra.run.smithi187.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-25T13:59:55.824 INFO:teuthology.orchestra.run.smithi150.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-04-25T13:59:55.842 INFO:teuthology.orchestra.run.smithi187.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-25T13:59:55.849 INFO:teuthology.orchestra.run.smithi150.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-04-25T13:59:55.868 INFO:teuthology.orchestra.run.smithi187.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 441 kB/s | 25 kB 00:00 2024-04-25T13:59:55.883 INFO:teuthology.orchestra.run.smithi150.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-04-25T13:59:55.893 INFO:teuthology.orchestra.run.smithi187.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-04-25T13:59:55.925 INFO:teuthology.orchestra.run.smithi150.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.6 MB/s | 272 kB 00:00 2024-04-25T13:59:55.926 INFO:teuthology.orchestra.run.smithi187.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-25T13:59:55.950 INFO:teuthology.orchestra.run.smithi150.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-04-25T13:59:55.951 INFO:teuthology.orchestra.run.smithi187.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-25T13:59:55.975 INFO:teuthology.orchestra.run.smithi150.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 974 kB/s | 90 kB 00:00 2024-04-25T13:59:55.977 INFO:teuthology.orchestra.run.smithi187.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 707 kB/s | 35 kB 00:00 2024-04-25T13:59:56.034 INFO:teuthology.orchestra.run.smithi150.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-25T13:59:56.060 INFO:teuthology.orchestra.run.smithi150.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 283 kB/s | 31 kB 00:00 2024-04-25T13:59:56.093 INFO:teuthology.orchestra.run.smithi150.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-25T13:59:56.143 INFO:teuthology.orchestra.run.smithi150.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-25T13:59:56.169 INFO:teuthology.orchestra.run.smithi150.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 329 kB/s | 36 kB 00:00 2024-04-25T13:59:56.194 INFO:teuthology.orchestra.run.smithi150.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 509 kB/s | 51 kB 00:00 2024-04-25T13:59:56.211 INFO:teuthology.orchestra.run.smithi187.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-25T13:59:56.228 INFO:teuthology.orchestra.run.smithi150.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-25T13:59:56.245 INFO:teuthology.orchestra.run.smithi150.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-04-25T13:59:56.295 INFO:teuthology.orchestra.run.smithi150.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-25T13:59:56.311 INFO:teuthology.orchestra.run.smithi187.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-25T13:59:56.329 INFO:teuthology.orchestra.run.smithi150.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 198 kB/s | 20 kB 00:00 2024-04-25T13:59:56.345 INFO:teuthology.orchestra.run.smithi187.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T13:59:56.354 INFO:teuthology.orchestra.run.smithi150.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-25T13:59:56.403 INFO:teuthology.orchestra.run.smithi187.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 9.9 MB/s | 911 kB 00:00 2024-04-25T13:59:56.428 INFO:teuthology.orchestra.run.smithi187.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-25T13:59:56.429 INFO:teuthology.orchestra.run.smithi150.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-04-25T13:59:56.545 INFO:teuthology.orchestra.run.smithi187.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 8.7 MB/s | 1.0 MB 00:00 2024-04-25T13:59:56.646 INFO:teuthology.orchestra.run.smithi150.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.2 MB/s | 1.3 MB 00:00 2024-04-25T13:59:56.713 INFO:teuthology.orchestra.run.smithi187.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-04-25T13:59:56.721 INFO:teuthology.orchestra.run.smithi150.stdout:(150/152): librados2-19.0.0-3291.g994e2f20.el9. 8.5 MB/s | 3.3 MB 00:00 2024-04-25T13:59:56.737 INFO:teuthology.orchestra.run.smithi187.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 457 kB/s | 87 kB 00:00 2024-04-25T13:59:56.880 INFO:teuthology.orchestra.run.smithi150.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-04-25T13:59:56.938 INFO:teuthology.orchestra.run.smithi187.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-04-25T13:59:56.963 INFO:teuthology.orchestra.run.smithi187.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.1 MB/s | 244 kB 00:00 2024-04-25T13:59:56.988 INFO:teuthology.orchestra.run.smithi187.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T13:59:57.013 INFO:teuthology.orchestra.run.smithi187.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 514 kB/s | 26 kB 00:00 2024-04-25T13:59:57.039 INFO:teuthology.orchestra.run.smithi187.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-25T13:59:57.064 INFO:teuthology.orchestra.run.smithi187.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T13:59:57.090 INFO:teuthology.orchestra.run.smithi187.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 695 kB/s | 35 kB 00:00 2024-04-25T13:59:57.248 INFO:teuthology.orchestra.run.smithi187.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-04-25T13:59:57.273 INFO:teuthology.orchestra.run.smithi187.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-04-25T13:59:57.299 INFO:teuthology.orchestra.run.smithi187.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-25T13:59:57.324 INFO:teuthology.orchestra.run.smithi187.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T13:59:57.349 INFO:teuthology.orchestra.run.smithi187.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-25T13:59:57.374 INFO:teuthology.orchestra.run.smithi187.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T13:59:57.400 INFO:teuthology.orchestra.run.smithi187.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T13:59:57.425 INFO:teuthology.orchestra.run.smithi187.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T13:59:57.442 INFO:teuthology.orchestra.run.smithi187.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-25T13:59:57.459 INFO:teuthology.orchestra.run.smithi187.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.6 MB/s | 54 kB 00:00 2024-04-25T13:59:57.634 INFO:teuthology.orchestra.run.smithi187.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 238 kB/s | 42 kB 00:00 2024-04-25T13:59:57.776 INFO:teuthology.orchestra.run.smithi187.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 316 kB/s | 44 kB 00:00 2024-04-25T13:59:57.985 INFO:teuthology.orchestra.run.smithi187.stdout:(93/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-04-25T13:59:58.010 INFO:teuthology.orchestra.run.smithi187.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 950 kB/s | 221 kB 00:00 2024-04-25T13:59:58.035 INFO:teuthology.orchestra.run.smithi187.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 723 kB/s | 36 kB 00:00 2024-04-25T13:59:58.061 INFO:teuthology.orchestra.run.smithi187.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 945 kB/s | 47 kB 00:00 2024-04-25T13:59:58.094 INFO:teuthology.orchestra.run.smithi187.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-04-25T13:59:58.120 INFO:teuthology.orchestra.run.smithi187.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 981 kB/s | 58 kB 00:00 2024-04-25T13:59:58.153 INFO:teuthology.orchestra.run.smithi187.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-04-25T13:59:58.178 INFO:teuthology.orchestra.run.smithi187.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-04-25T13:59:58.204 INFO:teuthology.orchestra.run.smithi187.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 892 kB/s | 45 kB 00:00 2024-04-25T13:59:58.229 INFO:teuthology.orchestra.run.smithi187.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 445 kB/s | 22 kB 00:00 2024-04-25T13:59:58.262 INFO:teuthology.orchestra.run.smithi187.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 14 MB/s | 863 kB 00:00 2024-04-25T13:59:58.730 INFO:teuthology.orchestra.run.smithi150.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 3.1 MB 00:02 2024-04-25T13:59:58.737 INFO:teuthology.orchestra.run.smithi150.stdout:-------------------------------------------------------------------------------- 2024-04-25T13:59:58.738 INFO:teuthology.orchestra.run.smithi150.stdout:Total 15 MB/s | 214 MB 00:14 2024-04-25T13:59:58.913 INFO:teuthology.orchestra.run.smithi187.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 475 kB/s | 309 kB 00:00 2024-04-25T13:59:58.938 INFO:teuthology.orchestra.run.smithi187.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 775 kB/s | 548 kB 00:00 2024-04-25T13:59:59.147 INFO:teuthology.orchestra.run.smithi187.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-04-25T13:59:59.322 INFO:teuthology.orchestra.run.smithi187.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-04-25T13:59:59.472 INFO:teuthology.orchestra.run.smithi187.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 324 kB/s | 49 kB 00:00 2024-04-25T13:59:59.531 INFO:teuthology.orchestra.run.smithi187.stdout:(109/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-04-25T13:59:59.606 INFO:teuthology.orchestra.run.smithi187.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 504 kB/s | 67 kB 00:00 2024-04-25T13:59:59.664 INFO:teuthology.orchestra.run.smithi187.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-04-25T13:59:59.813 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction check 2024-04-25T13:59:59.823 INFO:teuthology.orchestra.run.smithi187.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-04-25T13:59:59.958 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction check succeeded. 2024-04-25T13:59:59.958 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction test 2024-04-25T13:59:59.982 INFO:teuthology.orchestra.run.smithi187.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-04-25T13:59:59.999 INFO:teuthology.orchestra.run.smithi187.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 247 kB/s | 43 kB 00:00 2024-04-25T14:00:00.124 INFO:teuthology.orchestra.run.smithi187.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 228 kB/s | 32 kB 00:00 2024-04-25T14:00:00.141 INFO:teuthology.orchestra.run.smithi187.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 106 kB/s | 15 kB 00:00 2024-04-25T14:00:00.250 INFO:teuthology.orchestra.run.smithi187.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2024-04-25T14:00:00.283 INFO:teuthology.orchestra.run.smithi187.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-04-25T14:00:00.400 INFO:teuthology.orchestra.run.smithi187.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 303 kB/s | 35 kB 00:00 2024-04-25T14:00:00.475 INFO:teuthology.orchestra.run.smithi187.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-25T14:00:00.567 INFO:teuthology.orchestra.run.smithi187.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.2 MB/s | 206 kB 00:00 2024-04-25T14:00:00.709 INFO:teuthology.orchestra.run.smithi187.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-25T14:00:00.734 INFO:teuthology.orchestra.run.smithi187.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 866 kB/s | 144 kB 00:00 2024-04-25T14:00:00.985 INFO:teuthology.orchestra.run.smithi187.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 39 kB/s | 11 kB 00:00 2024-04-25T14:00:01.010 INFO:teuthology.orchestra.run.smithi187.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 64 kB/s | 18 kB 00:00 2024-04-25T14:00:01.085 INFO:teuthology.orchestra.run.smithi187.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-04-25T14:00:01.177 INFO:teuthology.orchestra.run.smithi187.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 215 kB/s | 20 kB 00:00 2024-04-25T14:00:01.213 INFO:teuthology.orchestra.run.smithi187.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 96 kB/s | 19 kB 00:00 2024-04-25T14:00:01.313 INFO:teuthology.orchestra.run.smithi187.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-04-25T14:00:01.555 INFO:teuthology.orchestra.run.smithi187.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 2.7 MB/s | 1.0 MB 00:00 2024-04-25T14:00:01.580 INFO:teuthology.orchestra.run.smithi187.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 296 kB/s | 79 kB 00:00 2024-04-25T14:00:01.682 INFO:teuthology.orchestra.run.smithi150.stdout:Transaction test succeeded. 2024-04-25T14:00:01.682 INFO:teuthology.orchestra.run.smithi150.stdout:Running transaction 2024-04-25T14:00:01.722 INFO:teuthology.orchestra.run.smithi187.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 346 kB/s | 58 kB 00:00 2024-04-25T14:00:01.747 INFO:teuthology.orchestra.run.smithi187.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-04-25T14:00:01.948 INFO:teuthology.orchestra.run.smithi187.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 73 kB/s | 16 kB 00:00 2024-04-25T14:00:01.973 INFO:teuthology.orchestra.run.smithi187.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 400 kB/s | 90 kB 00:00 2024-04-25T14:00:02.073 INFO:teuthology.orchestra.run.smithi187.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-04-25T14:00:02.098 INFO:teuthology.orchestra.run.smithi187.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-04-25T14:00:02.190 INFO:teuthology.orchestra.run.smithi187.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-04-25T14:00:02.282 INFO:teuthology.orchestra.run.smithi187.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-04-25T14:00:02.332 INFO:teuthology.orchestra.run.smithi187.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 252 kB/s | 36 kB 00:00 2024-04-25T14:00:02.433 INFO:teuthology.orchestra.run.smithi187.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 339 kB/s | 51 kB 00:00 2024-04-25T14:00:02.458 INFO:teuthology.orchestra.run.smithi187.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-25T14:00:02.650 INFO:teuthology.orchestra.run.smithi187.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 414 kB/s | 90 kB 00:00 2024-04-25T14:00:02.775 INFO:teuthology.orchestra.run.smithi187.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 1.3 MB/s | 427 kB 00:00 2024-04-25T14:00:02.800 INFO:teuthology.orchestra.run.smithi187.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-04-25T14:00:02.959 INFO:teuthology.orchestra.run.smithi187.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-04-25T14:00:03.134 INFO:teuthology.orchestra.run.smithi187.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-04-25T14:00:03.301 INFO:teuthology.orchestra.run.smithi187.stdout:(148/152): librados2-19.0.0-3291.g994e2f20.el9. 9.7 MB/s | 3.3 MB 00:00 2024-04-25T14:00:03.385 INFO:teuthology.orchestra.run.smithi187.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 16 MB/s | 1.3 MB 00:00 2024-04-25T14:00:03.485 INFO:teuthology.orchestra.run.smithi187.stdout:(150/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-04-25T14:00:03.544 INFO:teuthology.orchestra.run.smithi187.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2024-04-25T14:00:07.828 INFO:teuthology.orchestra.run.smithi150.stdout: Preparing : 1/1 2024-04-25T14:00:07.917 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T14:00:07.956 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T14:00:08.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T14:00:08.043 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T14:00:08.068 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T14:00:08.110 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T14:00:08.438 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T14:00:08.561 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:00:08.686 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:00:08.712 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:00:08.795 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:00:08.825 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T14:00:08.971 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T14:00:09.021 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T14:00:09.049 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:00:09.154 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:00:09.184 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T14:00:09.246 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T14:00:09.268 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T14:00:09.304 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T14:00:09.452 INFO:teuthology.orchestra.run.smithi187.stdout:(152/152): ceph-test-19.0.0-3291.g994e2f20.el9. 2.7 MB/s | 43 MB 00:16 2024-04-25T14:00:09.452 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T14:00:09.457 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2024-04-25T14:00:09.457 INFO:teuthology.orchestra.run.smithi187.stdout:Total 10 MB/s | 214 MB 00:20 2024-04-25T14:00:09.484 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T14:00:09.591 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T14:00:09.687 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T14:00:09.725 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T14:00:09.760 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T14:00:09.832 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T14:00:09.867 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T14:00:09.910 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T14:00:09.943 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T14:00:09.972 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T14:00:10.041 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T14:00:10.106 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T14:00:10.324 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T14:00:10.377 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T14:00:10.453 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T14:00:10.487 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T14:00:10.530 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T14:00:10.559 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T14:00:10.573 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2024-04-25T14:00:10.629 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T14:00:10.674 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T14:00:10.715 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2024-04-25T14:00:10.715 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2024-04-25T14:00:10.748 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T14:00:10.891 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T14:00:10.943 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T14:00:11.461 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T14:00:11.555 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T14:00:11.584 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T14:00:11.626 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T14:00:11.665 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T14:00:11.707 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T14:00:11.859 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T14:00:11.917 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T14:00:11.972 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T14:00:12.046 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T14:00:12.076 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T14:00:12.113 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T14:00:12.160 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T14:00:12.196 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T14:00:12.239 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T14:00:12.281 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T14:00:12.418 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T14:00:12.436 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2024-04-25T14:00:12.436 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2024-04-25T14:00:12.470 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T14:00:12.500 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T14:00:12.542 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T14:00:12.636 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T14:00:12.667 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T14:00:12.707 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T14:00:12.732 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T14:00:12.757 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T14:00:12.831 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T14:00:12.959 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T14:00:12.998 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T14:00:13.026 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T14:00:13.060 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T14:00:13.099 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T14:00:13.131 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T14:00:13.161 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T14:00:13.195 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T14:00:13.229 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T14:00:13.270 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T14:00:13.305 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T14:00:13.383 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T14:00:13.426 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T14:00:13.458 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T14:00:13.505 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T14:00:13.604 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T14:00:13.651 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T14:00:14.374 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T14:00:14.423 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T14:00:14.457 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T14:00:14.555 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T14:00:14.615 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T14:00:14.702 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T14:00:14.735 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T14:00:14.906 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T14:00:14.932 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T14:00:15.027 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T14:00:15.672 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T14:00:15.780 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T14:00:17.008 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T14:00:17.094 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T14:00:17.122 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T14:00:17.156 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T14:00:17.476 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T14:00:17.546 INFO:teuthology.orchestra.run.smithi150.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T14:00:17.643 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T14:00:17.673 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T14:00:17.710 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T14:00:18.201 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T14:00:18.230 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T14:00:18.286 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2024-04-25T14:00:18.300 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T14:00:18.323 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T14:00:18.372 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T14:00:18.410 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T14:00:18.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T14:00:18.498 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T14:00:18.523 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T14:00:18.565 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T14:00:18.891 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T14:00:18.998 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:00:19.124 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:00:19.150 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:00:19.241 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:00:19.270 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T14:00:19.419 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T14:00:19.468 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T14:00:19.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:00:19.584 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:00:19.613 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T14:00:19.659 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T14:00:19.681 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T14:00:19.705 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T14:00:19.906 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T14:00:19.938 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T14:00:20.045 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T14:00:20.132 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T14:00:20.171 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T14:00:20.206 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T14:00:20.270 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T14:00:20.313 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T14:00:20.356 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T14:00:20.389 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T14:00:20.425 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T14:00:20.493 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T14:00:20.535 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:20.556 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T14:00:20.569 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:20.586 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T14:00:20.629 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T14:00:20.705 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T14:00:20.740 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T14:00:20.784 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T14:00:20.812 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T14:00:20.883 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T14:00:20.944 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T14:00:21.018 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T14:00:21.150 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T14:00:21.163 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:21.204 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T14:00:21.223 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T14:00:21.327 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T14:00:21.328 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T14:00:21.328 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T14:00:21.328 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:21.358 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T14:00:21.723 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T14:00:21.808 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T14:00:21.837 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T14:00:21.880 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T14:00:21.918 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T14:00:21.959 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T14:00:22.107 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T14:00:22.171 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T14:00:22.226 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T14:00:22.298 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T14:00:22.330 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T14:00:22.366 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T14:00:22.412 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T14:00:22.449 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T14:00:22.509 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T14:00:22.551 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T14:00:22.686 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T14:00:22.731 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T14:00:22.761 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T14:00:22.803 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T14:00:22.897 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T14:00:22.936 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T14:00:22.976 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T14:00:23.002 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T14:00:23.026 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T14:00:23.101 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T14:00:23.227 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T14:00:23.267 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T14:00:23.295 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T14:00:23.329 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T14:00:23.368 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T14:00:23.399 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T14:00:23.429 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T14:00:23.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T14:00:23.497 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T14:00:23.538 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T14:00:23.573 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T14:00:23.659 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T14:00:23.712 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T14:00:23.743 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T14:00:23.790 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T14:00:23.888 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T14:00:23.936 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T14:00:24.660 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T14:00:24.708 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T14:00:24.741 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T14:00:24.839 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T14:00:24.891 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T14:00:24.987 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T14:00:25.102 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T14:00:25.277 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T14:00:25.307 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T14:00:25.385 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T14:00:26.025 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T14:00:26.138 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T14:00:27.359 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T14:00:27.453 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T14:00:27.481 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T14:00:27.514 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T14:00:27.832 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T14:00:27.877 INFO:teuthology.orchestra.run.smithi187.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T14:00:27.967 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T14:00:27.998 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T14:00:28.034 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T14:00:28.519 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T14:00:28.554 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T14:00:28.609 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T14:00:28.631 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T14:00:30.847 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:30.886 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:31.475 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T14:00:31.514 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T14:00:31.606 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T14:00:31.607 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T14:00:31.607 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T14:00:31.607 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:31.640 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /sys 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /proc 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /mnt 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /var/tmp 2024-04-25T14:00:32.983 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /home 2024-04-25T14:00:32.984 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /root 2024-04-25T14:00:32.984 INFO:teuthology.orchestra.run.smithi150.stdout:skipping the directory /tmp 2024-04-25T14:00:32.984 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:33.154 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T14:00:33.169 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T14:00:33.169 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:33.169 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T14:00:33.169 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T14:00:34.765 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T14:00:34.765 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:34.765 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T14:00:34.765 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T14:00:34.766 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T14:00:34.766 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:35.007 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T14:00:35.029 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:35.491 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T14:00:35.513 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:35.621 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T14:00:35.678 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T14:00:35.734 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T14:00:35.921 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T14:00:35.967 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T14:00:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T14:00:36.135 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T14:00:36.272 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T14:00:36.317 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T14:00:36.349 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T14:00:36.376 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T14:00:36.762 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T14:00:36.800 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T14:00:36.824 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T14:00:36.877 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T14:00:36.906 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T14:00:37.947 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T14:00:37.975 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T14:00:38.946 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T14:00:38.975 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T14:00:39.076 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T14:00:39.241 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T14:00:39.269 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T14:00:39.292 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:39.336 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T14:00:39.348 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T14:00:39.446 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T14:00:39.494 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T14:00:40.428 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T14:00:40.455 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T14:00:40.477 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T14:00:40.477 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:40.477 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T14:00:40.477 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T14:00:40.478 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T14:00:40.478 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:40.523 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T14:00:40.545 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T14:00:40.545 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:40.545 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T14:00:40.545 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:40.590 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T14:00:40.613 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:40.932 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T14:00:40.955 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T14:00:40.956 INFO:teuthology.orchestra.run.smithi150.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:40.956 INFO:teuthology.orchestra.run.smithi150.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T14:00:40.956 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T14:00:40.956 INFO:teuthology.orchestra.run.smithi150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T14:00:40.956 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:43.146 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T14:00:43.147 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /sys 2024-04-25T14:00:43.147 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /proc 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /mnt 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /var/tmp 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /home 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /root 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout:skipping the directory /tmp 2024-04-25T14:00:43.148 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:43.327 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T14:00:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T14:00:43.342 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:43.342 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T14:00:43.342 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:44.869 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T14:00:44.895 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:45.120 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T14:00:45.142 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T14:00:45.142 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:45.142 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T14:00:45.143 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T14:00:45.143 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T14:00:45.143 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:45.237 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T14:00:45.278 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T14:00:45.318 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T14:00:45.350 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T14:00:45.377 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T14:00:45.563 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T14:00:45.604 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T14:00:45.607 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T14:00:45.626 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T14:00:45.627 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:45.627 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T14:00:45.627 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T14:00:45.627 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T14:00:45.627 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:45.637 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T14:00:45.725 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T14:00:45.773 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T14:00:45.829 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T14:00:46.023 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T14:00:46.070 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T14:00:46.139 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T14:00:46.227 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T14:00:46.285 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:46.326 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:46.345 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:46.366 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T14:00:46.404 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T14:00:46.436 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T14:00:46.464 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T14:00:46.916 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T14:00:46.945 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T14:00:46.977 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T14:00:47.039 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T14:00:47.077 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T14:00:47.183 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T14:00:47.256 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T14:00:47.297 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T14:00:47.378 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T14:00:47.418 INFO:teuthology.orchestra.run.smithi150.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T14:00:47.419 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T14:00:47.494 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T14:00:47.535 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T14:00:47.536 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T14:00:47.633 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T14:00:47.701 INFO:teuthology.orchestra.run.smithi150.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:00:48.109 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T14:00:48.146 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T14:00:49.104 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T14:00:49.129 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T14:00:49.230 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T14:00:49.395 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T14:00:49.431 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T14:00:49.454 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:49.498 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T14:00:49.510 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T14:00:49.601 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T14:00:49.655 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T14:00:50.585 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T14:00:50.626 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T14:00:50.649 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T14:00:50.649 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:50.649 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T14:00:50.649 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T14:00:50.649 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T14:00:50.650 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:50.693 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T14:00:50.715 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T14:00:50.715 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:50.716 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T14:00:50.716 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:50.761 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T14:00:50.784 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T14:00:50.784 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:50.784 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T14:00:50.784 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T14:00:50.785 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T14:00:50.785 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:51.095 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T14:00:51.118 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T14:00:51.118 INFO:teuthology.orchestra.run.smithi187.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T14:00:51.118 INFO:teuthology.orchestra.run.smithi187.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T14:00:51.119 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T14:00:51.119 INFO:teuthology.orchestra.run.smithi187.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T14:00:51.119 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:00:53.824 INFO:teuthology.orchestra.run.smithi150.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:00:53.824 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T14:00:53.824 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T14:00:53.824 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T14:00:53.825 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T14:00:53.826 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T14:00:53.838 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T14:00:53.839 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T14:00:53.840 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T14:00:53.841 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T14:00:53.842 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T14:00:53.843 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T14:00:53.844 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T14:00:53.845 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T14:00:53.846 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T14:00:53.847 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T14:00:55.373 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T14:00:55.414 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T14:00:55.454 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T14:00:55.487 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T14:00:55.514 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T14:00:55.700 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T14:00:55.752 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T14:00:55.782 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout:Upgraded: 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout:Installed: 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T14:00:55.932 INFO:teuthology.orchestra.run.smithi150.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.933 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T14:00:55.934 INFO:teuthology.orchestra.run.smithi150.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.935 INFO:teuthology.orchestra.run.smithi150.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T14:00:55.936 INFO:teuthology.orchestra.run.smithi150.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T14:00:55.937 INFO:teuthology.orchestra.run.smithi150.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T14:00:55.938 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T14:00:55.939 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T14:00:55.940 INFO:teuthology.orchestra.run.smithi150.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T14:00:55.941 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T14:00:55.942 INFO:teuthology.orchestra.run.smithi150.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T14:00:55.943 INFO:teuthology.orchestra.run.smithi150.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T14:00:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T14:00:55.944 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:00:55.944 INFO:teuthology.orchestra.run.smithi150.stdout:Complete! 2024-04-25T14:00:56.317 DEBUG:teuthology.parallel:result is None 2024-04-25T14:00:56.361 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:56.404 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:56.424 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T14:00:57.256 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T14:00:57.343 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T14:00:57.382 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T14:00:57.463 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T14:00:57.496 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T14:00:57.496 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T14:00:57.580 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T14:00:57.613 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T14:00:57.614 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T14:00:57.697 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T14:00:57.763 INFO:teuthology.orchestra.run.smithi187.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T14:01:03.830 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T14:01:03.831 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T14:01:03.832 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T14:01:03.832 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T14:01:03.832 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T14:01:03.832 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T14:01:03.832 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T14:01:03.844 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T14:01:03.845 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T14:01:03.846 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T14:01:03.847 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T14:01:03.848 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T14:01:03.849 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T14:01:03.850 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T14:01:03.851 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T14:01:03.852 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T14:01:03.853 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout:Upgraded: 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T14:01:05.967 INFO:teuthology.orchestra.run.smithi187.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.968 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T14:01:05.969 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T14:01:05.970 INFO:teuthology.orchestra.run.smithi187.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T14:01:05.971 INFO:teuthology.orchestra.run.smithi187.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T14:01:05.972 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T14:01:05.973 INFO:teuthology.orchestra.run.smithi187.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T14:01:05.974 INFO:teuthology.orchestra.run.smithi187.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T14:01:05.975 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T14:01:05.976 INFO:teuthology.orchestra.run.smithi187.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T14:01:05.977 INFO:teuthology.orchestra.run.smithi187.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T14:01:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T14:01:05.978 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:05.978 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2024-04-25T14:01:06.384 DEBUG:teuthology.parallel:result is None 2024-04-25T14:01:06.385 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T14:01:06.603 DEBUG:teuthology.orchestra.run.smithi150:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T14:01:06.628 INFO:teuthology.orchestra.run.smithi150.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T14:01:06.629 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T14:01:06.629 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T14:01:06.631 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T14:01:06.846 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T14:01:06.871 INFO:teuthology.orchestra.run.smithi187.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T14:01:06.872 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T14:01:06.872 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T14:01:06.874 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T14:01:06.874 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:06.874 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T14:01:06.904 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:06.904 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T14:01:06.939 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T14:01:06.939 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:06.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T14:01:06.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T14:01:07.040 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:07.040 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T14:01:07.064 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T14:01:07.129 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T14:01:07.129 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:07.129 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T14:01:07.156 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T14:01:07.222 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:07.222 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T14:01:07.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T14:01:07.308 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T14:01:07.309 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:07.309 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T14:01:07.334 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T14:01:07.399 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:07.399 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T14:01:07.424 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T14:01:07.488 INFO:teuthology.run_tasks:Running task ceph... 2024-04-25T14:01:07.573 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-25T14:01:07.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 777 /var/log/ceph 2024-04-25T14:01:07.575 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 777 /var/log/ceph 2024-04-25T14:01:07.599 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-25T14:01:07.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T14:01:07.640 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-25T14:01:07.664 INFO:tasks.ceph:Creating extra log directories... 2024-04-25T14:01:07.665 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T14:01:07.706 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-25T14:01:07.736 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-25T14:01:07.736 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-25T14:01:07.736 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-25T14:01:07.736 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:07.736 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T14:01:07.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:07.890 DEBUG:teuthology.orchestra.run.smithi150:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:07.929 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:07.929 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-25T14:01:07.945 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:08.052 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:08.096 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-25T14:01:08.096 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-25T14:01:08.096 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193', 'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7673193', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240425.015853-debug'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240425.015853-debug', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1880, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'sleep_before_teardown': 0, 'subset': '178/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240425.015853-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi150.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBB1pqlg+w8go4L1U+8SvxXDeMrWNBQT2Q2eokT/agJHjTJ5ZqmemvfrXIl3V4jzsNgUbtcptsdy8RP6Y7HdO5ko=', 'smithi187.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL1nZRCwfzW400zFnxkxoap6HYcxL2bfWFlP+6ulqjlmO1QwT0SykYcj8x3+lWYxpuWdY6lSqeR6qcthHq4Y6Ec='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_openfiletable']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-25_10:00:23', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-25T14:01:08.097 DEBUG:teuthology.orchestra.run.smithi150:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T14:01:08.099 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-25T14:01:08.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T14:01:08.154 DEBUG:teuthology.orchestra.run.smithi187:> sudo install -d -m0777 -- /var/run/ceph 2024-04-25T14:01:08.210 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:08.210 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T14:01:08.230 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T14:01:08.230 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_1 2024-04-25T14:01:08.284 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-25 14:00:48.105494742 +0000 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-25 13:59:01.617241882 +0000 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-25 13:59:01.617241882 +0000 2024-04-25T14:01:08.285 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-25 13:59:01.617241882 +0000 2024-04-25T14:01:08.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T14:01:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:01:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:01:08.347 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000258485 s, 2.0 MB/s 2024-04-25T14:01:08.348 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T14:01:08.404 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_2 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-25 14:00:48.105494742 +0000 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-25 13:59:01.955234903 +0000 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-25 13:59:01.955234903 +0000 2024-04-25T14:01:08.460 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-25 13:59:01.955234903 +0000 2024-04-25T14:01:08.460 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T14:01:08.522 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:01:08.522 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:01:08.522 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000165187 s, 3.1 MB/s 2024-04-25T14:01:08.524 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T14:01:08.579 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_3 2024-04-25T14:01:08.633 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T14:01:08.633 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:08.633 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-25 14:00:48.106494723 +0000 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-25 13:59:02.302227738 +0000 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-25 13:59:02.302227738 +0000 2024-04-25T14:01:08.634 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-25 13:59:02.302227738 +0000 2024-04-25T14:01:08.634 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T14:01:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:01:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:01:08.696 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000161454 s, 3.2 MB/s 2024-04-25T14:01:08.697 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T14:01:08.752 DEBUG:teuthology.orchestra.run.smithi150:> stat /dev/vg_nvme/lv_4 2024-04-25T14:01:08.807 INFO:teuthology.orchestra.run.smithi150.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Access: 2024-04-25 14:00:48.106494723 +0000 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Modify: 2024-04-25 13:59:02.633220904 +0000 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout:Change: 2024-04-25 13:59:02.633220904 +0000 2024-04-25T14:01:08.808 INFO:teuthology.orchestra.run.smithi150.stdout: Birth: 2024-04-25 13:59:02.633220904 +0000 2024-04-25T14:01:08.808 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T14:01:08.869 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:01:08.870 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:01:08.870 INFO:teuthology.orchestra.run.smithi150.stderr:512 bytes copied, 0.000182506 s, 2.8 MB/s 2024-04-25T14:01:08.871 DEBUG:teuthology.orchestra.run.smithi150:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T14:01:08.926 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:08.926 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:08.926 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T14:01:08.942 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T14:01:08.942 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_1 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-25 14:00:58.127376038 +0000 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-25 13:59:01.635880152 +0000 2024-04-25T14:01:08.998 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-25 13:59:01.635880152 +0000 2024-04-25T14:01:08.999 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-25 13:59:01.635880152 +0000 2024-04-25T14:01:08.999 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T14:01:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-25T14:01:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-25T14:01:09.061 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000263586 s, 1.9 MB/s 2024-04-25T14:01:09.062 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T14:01:09.118 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_2 2024-04-25T14:01:09.173 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-25 14:00:58.127376038 +0000 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-25 13:59:01.978874231 +0000 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-25 13:59:01.978874231 +0000 2024-04-25T14:01:09.174 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-25 13:59:01.978874231 +0000 2024-04-25T14:01:09.174 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T14:01:09.236 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-25T14:01:09.236 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-25T14:01:09.236 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000173717 s, 2.9 MB/s 2024-04-25T14:01:09.238 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T14:01:09.293 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_3 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:09.348 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-25 14:00:58.127376038 +0000 2024-04-25T14:01:09.349 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-25 13:59:02.314868585 +0000 2024-04-25T14:01:09.349 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-25 13:59:02.314868585 +0000 2024-04-25T14:01:09.349 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-25 13:59:02.314868585 +0000 2024-04-25T14:01:09.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T14:01:09.411 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-25T14:01:09.411 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-25T14:01:09.411 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000194694 s, 2.6 MB/s 2024-04-25T14:01:09.412 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T14:01:09.468 DEBUG:teuthology.orchestra.run.smithi187:> stat /dev/vg_nvme/lv_4 2024-04-25T14:01:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T14:01:09.522 INFO:teuthology.orchestra.run.smithi187.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Access: 2024-04-25 14:00:58.128376022 +0000 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Modify: 2024-04-25 13:59:02.635863192 +0000 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout:Change: 2024-04-25 13:59:02.635863192 +0000 2024-04-25T14:01:09.523 INFO:teuthology.orchestra.run.smithi187.stdout: Birth: 2024-04-25 13:59:02.635863192 +0000 2024-04-25T14:01:09.523 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T14:01:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records in 2024-04-25T14:01:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:1+0 records out 2024-04-25T14:01:09.585 INFO:teuthology.orchestra.run.smithi187.stderr:512 bytes copied, 0.000240916 s, 2.1 MB/s 2024-04-25T14:01:09.586 DEBUG:teuthology.orchestra.run.smithi187:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T14:01:09.642 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:09.642 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T14:01:09.642 INFO:tasks.ceph:Generating config... 2024-04-25T14:01:09.643 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-25T14:01:09.644 INFO:tasks.ceph:[client] debug client = 20 2024-04-25T14:01:09.644 INFO:tasks.ceph:[client] debug ms = 1 2024-04-25T14:01:09.644 INFO:tasks.ceph:[client] pid file = 2024-04-25T14:01:09.644 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-25T14:01:09.644 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-25T14:01:09.644 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-25T14:01:09.644 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-25T14:01:09.645 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-25T14:01:09.645 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-25T14:01:09.646 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-25T14:01:09.646 INFO:tasks.ceph:Setting up mon.a... 2024-04-25T14:01:09.646 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-25T14:01:09.724 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.keyring 2024-04-25T14:01:09.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-25T14:01:09.777 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T14:01:09.800 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.150'), ('mon.b', '172.21.15.187'), ('mon.c', '[v2:172.21.15.187:3301,v1:172.21.15.187:6790]')] 2024-04-25T14:01:09.801 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-25T14:01:09.801 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:09.801 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T14:01:09.855 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.150 --add b 172.21.15.187 --addv c '[v2:172.21.15.187:3301,v1:172.21.15.187:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: generated fsid 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:01:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:setting min_mon_release = quincy 2024-04-25T14:01:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:epoch 0 2024-04-25T14:01:09.951 INFO:teuthology.orchestra.run.smithi150.stdout:fsid 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:last_changed 2024-04-25T14:01:09.953035+0000 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:created 2024-04-25T14:01:09.953035+0000 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:min_mon_release 17 (quincy) 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:election_strategy: 1 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:0: [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] mon.a 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:1: [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] mon.b 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:2: [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] mon.c 2024-04-25T14:01:09.952 INFO:teuthology.orchestra.run.smithi150.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-25T14:01:09.953 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-25T14:01:09.967 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 77163e42-55ae-4cab-b506-bd8c531cc087... 2024-04-25T14:01:09.968 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:01:10.009 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:01:10.047 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-25T14:01:10.048 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-25T14:01:10.049 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-25T14:01:10.050 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-04-25T14:01:10.051 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.052 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:01:10.053 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:01:10.054 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-25T14:01:10.055 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:01:10.056 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-04-25T14:01:10.057 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-04-25T14:01:10.058 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-04-25T14:01:10.059 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-04-25T14:01:10.059 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-04-25T14:01:10.059 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:01:10.059 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:01:10.059 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:01:10.060 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-25T14:01:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-25T14:01:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:01:10.060 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-25T14:01:10.061 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:01:10.062 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-25T14:01:10.063 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-25T14:01:10.064 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.065 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-25T14:01:10.066 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:01:10.067 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:01:10.068 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-25T14:01:10.069 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:01:10.070 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:01:10.071 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:01:10.071 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-25T14:01:10.072 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-25T14:01:10.163 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-25T14:01:10.163 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:10.163 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-25T14:01:10.177 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:10.178 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-25T14:01:10.232 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:01:10.233 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:10.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T14:01:10.233 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T14:01:10.305 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:10.305 DEBUG:teuthology.orchestra.run.smithi150:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:10.360 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:01:10.360 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:10.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-25T14:01:10.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-25T14:01:10.392 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:10.392 DEBUG:teuthology.orchestra.run.smithi187:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:10.447 INFO:tasks.ceph:Setting up mon nodes... 2024-04-25T14:01:10.447 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-25T14:01:10.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T14:01:10.505 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-25T14:01:10.508 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T14:01:10.598 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-25T14:01:10.601 INFO:tasks.ceph:Setting up mds nodes... 2024-04-25T14:01:10.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T14:01:10.659 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-25T14:01:10.662 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-25T14:01:10.685 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T14:01:10.783 INFO:teuthology.orchestra.run.smithi150.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-25T14:01:10.786 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-25T14:01:10.809 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T14:01:10.867 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-25T14:01:10.870 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-25T14:01:10.894 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T14:01:10.991 INFO:teuthology.orchestra.run.smithi187.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-25T14:01:10.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-25T14:01:11.017 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-25T14:01:11.017 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T14:01:11.066 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-25T14:01:11.079 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-25T14:01:11.127 INFO:teuthology.orchestra.run.smithi150.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-25T14:01:11.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-25T14:01:11.187 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-25T14:01:11.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-25T14:01:11.246 INFO:teuthology.orchestra.run.smithi187.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-25T14:01:11.259 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-25T14:01:11.259 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi150.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi187.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-25T14:01:11.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-25T14:01:11.283 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:11.283 INFO:tasks.ceph:role: osd.0 2024-04-25T14:01:11.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:01:11.283 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:11.355 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:11.356 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:11.356 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:11.836 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-25T14:01:11.837 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-25T14:01:11.838 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-25T14:01:11.948 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-25T14:01:11.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-25T14:01:12.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:12.042 INFO:tasks.ceph:role: osd.1 2024-04-25T14:01:12.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:01:12.042 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:12.113 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:12.114 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:12.114 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:12.114 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:12.572 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-25T14:01:12.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-25T14:01:12.575 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-25T14:01:12.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-25T14:01:12.726 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-25T14:01:12.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:12.791 INFO:tasks.ceph:role: osd.2 2024-04-25T14:01:12.791 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:01:12.792 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T14:01:12.863 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:12.863 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:12.863 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:12.863 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:12.864 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:13.352 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-25T14:01:13.354 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-25T14:01:13.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-25T14:01:13.466 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-25T14:01:13.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-25T14:01:13.568 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:13.569 INFO:tasks.ceph:role: osd.3 2024-04-25T14:01:13.569 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:01:13.569 DEBUG:teuthology.orchestra.run.smithi150:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T14:01:13.643 INFO:teuthology.orchestra.run.smithi150.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:13.644 INFO:teuthology.orchestra.run.smithi150.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:13.645 INFO:teuthology.orchestra.run.smithi150.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:13.645 INFO:teuthology.orchestra.run.smithi150.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:13.645 INFO:teuthology.orchestra.run.smithi150.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:14.149 INFO:teuthology.orchestra.run.smithi150.stdout:Discarding blocks...Done. 2024-04-25T14:01:14.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi150.front.sepia.ceph.com -o noatime 2024-04-25T14:01:14.151 DEBUG:teuthology.orchestra.run.smithi150:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-25T14:01:14.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-25T14:01:14.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:14.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.437+0000 7fdfcc40f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-25T14:01:14.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.440+0000 7fdfcc40f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-25T14:01:14.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.440+0000 7fdfcc40f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T14:01:14.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.440+0000 7fdfcc40f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T14:01:14.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.440+0000 7fdfcc40f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-25T14:01:14.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:14.449+0000 7fdfcc40f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-25T14:01:17.288 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-25T14:01:17.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:17.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.414+0000 7fec08c1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-25T14:01:17.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.415+0000 7fec08c1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-25T14:01:17.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.415+0000 7fec08c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T14:01:17.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.415+0000 7fec08c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T14:01:17.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.415+0000 7fec08c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-25T14:01:17.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:17.415+0000 7fec08c1e640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-25T14:01:20.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-25T14:01:20.294 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.394+0000 7f8b1a3b6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.395+0000 7f8b1a3b6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.395+0000 7f8b1a3b6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.395+0000 7f8b1a3b6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.395+0000 7f8b1a3b6640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-25T14:01:20.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:20.395+0000 7f8b1a3b6640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-25T14:01:23.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-25T14:01:23.278 DEBUG:teuthology.orchestra.run.smithi150:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.378+0000 7efc7720f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.378+0000 7efc7720f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.378+0000 7efc7720f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.378+0000 7efc7720f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.378+0000 7efc7720f640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-25T14:01:23.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:23.379+0000 7efc7720f640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-25T14:01:26.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-25T14:01:26.262 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-25T14:01:26.292 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:26.292 INFO:tasks.ceph:role: osd.4 2024-04-25T14:01:26.292 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:01:26.292 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-25T14:01:26.366 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:26.366 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:26.366 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:26.366 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:26.367 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:26.880 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-25T14:01:26.882 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-25T14:01:26.882 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-25T14:01:27.011 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-25T14:01:27.045 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-25T14:01:27.111 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:27.111 INFO:tasks.ceph:role: osd.5 2024-04-25T14:01:27.111 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:01:27.111 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:27.185 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:27.641 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-25T14:01:27.643 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-25T14:01:27.644 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-25T14:01:27.726 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-25T14:01:27.794 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-25T14:01:27.858 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:27.858 INFO:tasks.ceph:role: osd.6 2024-04-25T14:01:27.858 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:01:27.858 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:27.928 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:28.410 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-25T14:01:28.412 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-25T14:01:28.412 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-25T14:01:28.536 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-25T14:01:28.566 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-25T14:01:28.630 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-25T14:01:28.630 INFO:tasks.ceph:role: osd.7 2024-04-25T14:01:28.631 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:01:28.631 DEBUG:teuthology.orchestra.run.smithi187:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout: = sunit=0 swidth=0 blks 2024-04-25T14:01:28.701 INFO:teuthology.orchestra.run.smithi187.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-25T14:01:28.702 INFO:teuthology.orchestra.run.smithi187.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-25T14:01:28.702 INFO:teuthology.orchestra.run.smithi187.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-25T14:01:28.702 INFO:teuthology.orchestra.run.smithi187.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-25T14:01:29.216 INFO:teuthology.orchestra.run.smithi187.stdout:Discarding blocks...Done. 2024-04-25T14:01:29.218 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi187.front.sepia.ceph.com -o noatime 2024-04-25T14:01:29.219 DEBUG:teuthology.orchestra.run.smithi187:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-25T14:01:29.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-25T14:01:29.398 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.506+0000 7fc532019640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-25T14:01:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.506+0000 7fc532019640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-25T14:01:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.506+0000 7fc532019640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T14:01:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.506+0000 7fc532019640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T14:01:29.506 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.506+0000 7fc532019640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-25T14:01:29.507 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:29.507+0000 7fc532019640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-25T14:01:32.360 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-25T14:01:32.388 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:32.491 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.491+0000 7fc326230640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-25T14:01:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.491+0000 7fc326230640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-25T14:01:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.491+0000 7fc326230640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T14:01:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.491+0000 7fc326230640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T14:01:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.491+0000 7fc326230640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-25T14:01:32.492 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:32.492+0000 7fc326230640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-25T14:01:35.340 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-25T14:01:35.367 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.467+0000 7f9623015640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-25T14:01:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.467+0000 7f9623015640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-25T14:01:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.467+0000 7f9623015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T14:01:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.467+0000 7f9623015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T14:01:35.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.467+0000 7f9623015640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-25T14:01:35.468 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:35.468+0000 7f9623015640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-25T14:01:38.098 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:38.101 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:01:38.324 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-25T14:01:38.349 DEBUG:teuthology.orchestra.run.smithi187:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.447+0000 7f485d501640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-25T14:01:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.448+0000 7f485d501640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-25T14:01:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.448+0000 7f485d501640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T14:01:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.448+0000 7f485d501640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T14:01:38.448 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.448+0000 7f485d501640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-25T14:01:38.449 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:38.448+0000 7f485d501640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-25T14:01:41.298 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-25T14:01:41.327 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-25T14:01:41.327 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.327 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-25T14:01:41.356 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-25T14:01:41.424 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.424 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-25T14:01:41.492 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-25T14:01:41.559 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.560 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-25T14:01:41.627 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-25T14:01:41.693 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:41.693 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-25T14:01:41.759 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:41.759 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-25T14:01:41.787 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:41.787 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-25T14:01:41.855 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:41.855 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-25T14:01:41.922 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:41.922 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-25T14:01:41.988 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:41.988 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-25T14:01:42.053 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:42.054 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-25T14:01:42.117 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:42.117 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-25T14:01:42.181 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:42.181 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-25T14:01:42.198 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:42.198 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-25T14:01:42.256 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:42.256 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-25T14:01:42.270 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:42.270 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-25T14:01:42.325 INFO:tasks.ceph:Adding keys to all mons... 2024-04-25T14:01:42.325 DEBUG:teuthology.orchestra.run.smithi150:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T14:01:42.327 DEBUG:teuthology.orchestra.run.smithi187:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.y] 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmdMg8HhAAGiDa9V7HIfyPF1HvP9u4rQ== 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmu4BmJxAA97zMMh3D3KUNtsDbGwkzDA== 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmGRrILhAAIg2pUAKfYtkPHonZ2jFm2Q== 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.0] 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAqYipma5wlGhAA4H2WY5d5faQLkw54oTdhvg== 2024-04-25T14:01:42.369 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.1] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAtYipm9+3RGBAANzSmNtvSkPp3PZqNPOc5hQ== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.2] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAwYipmb72gFxAAXszjhp98sXfU3mERSpXVYA== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.3] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAzYipmN9yoFhAAUeT/1IDgSofY9Txb6wBIoA== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr.x] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmoMzCIxAAMQ8RkeGQMgn4UuygerC8eg== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmwLnMMxAATYxzVsSAZZUjQJuFzeuaSA== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAmYipmwO4tOxAA8v8nYbz09RgqeKeG1kP8Aw== 2024-04-25T14:01:42.370 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.4] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA5YipmorY9HhAAStUVEPsdQwhrk3CRCyKReg== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.5] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA8YipmFsVgHRAAuxUDVhP1YYbOIuqKnk/0rg== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.6] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQA/YipmCeXsGxAAMKhPXK0qkbkSvZjER5hCxA== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[osd.7] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQBCYipmfInFGhAAQzv1lzLJbEZsVlzxNER6xw== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[client.2] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipm2NSoBxAABm1fUKNHFMNuiHMmSMrKlA== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[client.1] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipmDUJACxAAcFyylAXclzW4D6C20RFk4Q== 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout:[client.3] 2024-04-25T14:01:42.371 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipmm2fQDhAANojc8uxJB15T5dVpgYA4Vg== 2024-04-25T14:01:42.387 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.y] 2024-04-25T14:01:42.387 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmdMg8HhAAGiDa9V7HIfyPF1HvP9u4rQ== 2024-04-25T14:01:42.387 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmu4BmJxAA97zMMh3D3KUNtsDbGwkzDA== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmGRrILhAAIg2pUAKfYtkPHonZ2jFm2Q== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.0] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAqYipma5wlGhAA4H2WY5d5faQLkw54oTdhvg== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.1] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAtYipm9+3RGBAANzSmNtvSkPp3PZqNPOc5hQ== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.2] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAwYipmb72gFxAAXszjhp98sXfU3mERSpXVYA== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.3] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAzYipmN9yoFhAAUeT/1IDgSofY9Txb6wBIoA== 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr.x] 2024-04-25T14:01:42.388 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmoMzCIxAAMQ8RkeGQMgn4UuygerC8eg== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmwLnMMxAATYxzVsSAZZUjQJuFzeuaSA== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAmYipmwO4tOxAA8v8nYbz09RgqeKeG1kP8Aw== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.4] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA5YipmorY9HhAAStUVEPsdQwhrk3CRCyKReg== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.5] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA8YipmFsVgHRAAuxUDVhP1YYbOIuqKnk/0rg== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.6] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQA/YipmCeXsGxAAMKhPXK0qkbkSvZjER5hCxA== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[osd.7] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQBCYipmfInFGhAAQzv1lzLJbEZsVlzxNER6xw== 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout:[client.0] 2024-04-25T14:01:42.389 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ== 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout:[client.2] 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAnYipm2NSoBxAABm1fUKNHFMNuiHMmSMrKlA== 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout:[client.1] 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAnYipmDUJACxAAcFyylAXclzW4D6C20RFk4Q== 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout:[client.3] 2024-04-25T14:01:42.390 INFO:teuthology.orchestra.run.smithi187.stdout: key = AQAnYipmm2fQDhAANojc8uxJB15T5dVpgYA4Vg== 2024-04-25T14:01:42.391 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T14:01:42.416 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T14:01:42.490 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.492 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.557 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.617 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.620 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.681 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.742 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.800 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.802 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:42.860 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T14:01:42.862 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-25T14:01:42.921 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.923 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:42.986 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-25T14:01:43.047 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.050 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.169 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.229 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.232 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-25T14:01:43.290 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.293 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.354 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.357 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.419 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.421 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.485 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-25T14:01:43.547 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-25T14:01:43.548 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-25T14:01:43.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T14:01:43.678 DEBUG:teuthology.orchestra.run.smithi150:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-25T14:01:43.701 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-25T14:01:43.725 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T14:01:43.829 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-25T14:01:43.852 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-25T14:01:43.917 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-25T14:01:44.022 DEBUG:teuthology.orchestra.run.smithi187:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-25T14:01:44.045 DEBUG:teuthology.orchestra.run.smithi150:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:44.048 DEBUG:teuthology.orchestra.run.smithi187:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-25T14:01:44.100 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-25T14:01:44.100 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-25T14:01:44.101 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-25T14:01:44.103 INFO:tasks.ceph.mon.a:Started 2024-04-25T14:01:44.103 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-25T14:01:44.103 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-25T14:01:44.143 INFO:tasks.ceph.mon.b:Started 2024-04-25T14:01:44.143 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-25T14:01:44.143 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-25T14:01:44.145 INFO:tasks.ceph.mon.c:Started 2024-04-25T14:01:44.145 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-25T14:01:44.145 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-25T14:01:44.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-25T14:01:44.148 INFO:tasks.ceph.mgr.y:Started 2024-04-25T14:01:44.148 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-25T14:01:44.148 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-25T14:01:44.150 INFO:tasks.ceph.mgr.x:Started 2024-04-25T14:01:44.150 INFO:tasks.ceph:Setting crush tunables to default 2024-04-25T14:01:44.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph osd crush tunables default 2024-04-25T14:01:44.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.263+0000 7fc4ff306640 1 Processor -- start 2024-04-25T14:01:44.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.263+0000 7fc4ff306640 1 -- start start 2024-04-25T14:01:44.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4ff306640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4f8168c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4ff306640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f815e650 con 0x7fc4f80b33c0 2024-04-25T14:01:44.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4ff306640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4f8147af0 con 0x7fc4f8147610 2024-04-25T14:01:44.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4ff306640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f8147dd0 con 0x7fc4f80b3ee0 2024-04-25T14:01:44.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fc8a1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc4f80b33c0 0x7fc4f80b37d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33826/0 (socket says 172.21.15.150:33826) 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fc8a1640 1 -- 172.21.15.150:0/319398229 learned_addr learned my addr 172.21.15.150:0/319398229 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4f8168c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 msgr2=0x7fc4f8168c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 msgr2=0x7fc4f8168c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4f8168c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:01:44.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.265+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4f8168c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T14:01:44.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.427+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1549421539 0 0) 0x7fc4f815e650 con 0x7fc4f80b33c0 2024-04-25T14:01:44.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.427+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4e0003520 con 0x7fc4f80b33c0 2024-04-25T14:01:44.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1771891391 0 0) 0x7fc4e0003520 con 0x7fc4f80b33c0 2024-04-25T14:01:44.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 msgr2=0x7fc4f8168c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4f8168c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 >> v1:172.21.15.187:6789/0 conn(0x7fc4f8147610 legacy=0x7fc4f80b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f80b28e0 con 0x7fc4f80b33c0 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fc4ec001690 con 0x7fc4f80b33c0 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 >> v1:172.21.15.150:6789/0 conn(0x7fc4f80b33c0 legacy=0x7fc4f80b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4e00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.428+0000 7fc4f7fff640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e0004060 0x7fc4e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4f7fff640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e0004d90 0x7fc4e0005260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f8147dd0 con 0x7fc4e0004d90 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e000bc50 con 0x7fc4e0004060 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc4e000be50 con 0x7fc4f80b3ee0 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e0004d90 0x7fc4e0005260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fc8a1640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e0004060 0x7fc4e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd0a2640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4e00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 msgr2=0x7fc4e00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:44.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4f80b3ee0 0x7fc4e00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e0004060 msgr2=0x7fc4e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:44.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4e0004060 0x7fc4e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.429+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e0003520 con 0x7fc4e0004d90 2024-04-25T14:01:44.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.430+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e0004d90 0x7fc4e0005260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8074b00 tx=0x7fc4e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b2b4cfa42ec856 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:44.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.430+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8033070 con 0x7fc4e0004d90 2024-04-25T14:01:44.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.430+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4e802f400 con 0x7fc4e0004d90 2024-04-25T14:01:44.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.430+0000 7fc4f7fff640 1 -- 172.21.15.150:0/319398229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4e8009bc0 con 0x7fc4e0004d90 2024-04-25T14:01:44.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 -- 172.21.15.150:0/319398229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e0004d90 msgr2=0x7fc4e0005260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 --2- 172.21.15.150:0/319398229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4e0004d90 0x7fc4e0005260 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8074b00 tx=0x7fc4e0005830 comp rx=0 tx=0).stop 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4fd8a3640 1 -- 172.21.15.150:0/319398229 reap_dead start 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 -- 172.21.15.150:0/319398229 shutdown_connections 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 -- 172.21.15.150:0/319398229 >> 172.21.15.150:0/319398229 conn(0x7fc4f8073d00 msgr2=0x7fc4f809b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 -- 172.21.15.150:0/319398229 shutdown_connections 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.431+0000 7fc4ff306640 1 -- 172.21.15.150:0/319398229 wait complete. 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.433+0000 7fc4ff306640 1 Processor -- start 2024-04-25T14:01:44.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.433+0000 7fc4ff306640 1 -- start start 2024-04-25T14:01:44.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 0x7fc4f8142610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f81408c0 0x7fc4f8142b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80b42f0 con 0x7fc4f8160f50 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc4f80b2ea0 con 0x7fc4f81408c0 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4ff306640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc4f80aee70 con 0x7fc4e0004d90 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fc8a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 0x7fc4f8142610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fc8a1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 0x7fc4f8142610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51208/0 (socket says 172.21.15.150:51208) 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:43678/0 (socket says 172.21.15.150:43678) 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fc8a1640 1 -- 172.21.15.150:0/474287581 learned_addr learned my addr 172.21.15.150:0/474287581 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd8a3640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f81408c0 0x7fc4f8142b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 -- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f81408c0 msgr2=0x7fc4f8142b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f81408c0 0x7fc4f8142b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 -- 172.21.15.150:0/474287581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 msgr2=0x7fc4f8142610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 0x7fc4f8142610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.435+0000 7fc4fd0a2640 1 -- 172.21.15.150:0/474287581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4f8161740 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4fd0a2640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000c00 tx=0x7fc4f000e910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0019070 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f00052e0 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f0003830 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4f815af70 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc4f001c070 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4f815b530 con 0x7fc4e0004d90 2024-04-25T14:01:44.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.436+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc4f0014b10 con 0x7fc4e0004d90 2024-04-25T14:01:44.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.437+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4f8076a40 con 0x7fc4e0004d90 2024-04-25T14:01:44.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.440+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc4f0024070 con 0x7fc4e0004d90 2024-04-25T14:01:44.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.505+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc4f80b4a60 con 0x7fc4e0004d90 2024-04-25T14:01:44.507 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.509+0000 7fc4f5ffb640 1 -- 172.21.15.150:0/474287581 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fc4f0014d10 con 0x7fc4e0004d90 2024-04-25T14:01:44.508 INFO:teuthology.orchestra.run.smithi150.stderr:adjusted tunables profile to default 2024-04-25T14:01:44.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 msgr2=0x7fc4f8160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4f0000c00 tx=0x7fc4f000e910 comp rx=0 tx=0).stop 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 shutdown_connections 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc4f81408c0 0x7fc4f8142b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc4f8160f50 0x7fc4f8142610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 --2- 172.21.15.150:0/474287581 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc4e0004d90 0x7fc4f8160a10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 >> 172.21.15.150:0/474287581 conn(0x7fc4f8073d00 msgr2=0x7fc4f806fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 shutdown_connections 2024-04-25T14:01:44.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:44.510+0000 7fc4ff306640 1 -- 172.21.15.150:0/474287581 wait complete. 2024-04-25T14:01:44.524 INFO:tasks.ceph:check_enable_crimson: False 2024-04-25T14:01:44.524 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-25T14:01:44.524 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:44.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-25T14:01:44.548 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:44.547+0000 7fde84436040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T14:01:44.549 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:44.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-25T14:01:44.555 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:44.556+0000 7fc041636040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T14:01:44.579 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:44.579 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-25T14:01:44.642 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:01:44.642 DEBUG:teuthology.orchestra.run.smithi150:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-25T14:01:44.693 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:44.694+0000 7fde84436040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T14:01:44.705 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:44.705 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-25T14:01:44.711 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:44.710+0000 7fc041636040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T14:01:44.729 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:44.729 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-25T14:01:44.749 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:44.749+0000 7fde84436040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T14:01:44.764 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:44.765+0000 7fc041636040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T14:01:44.772 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:44.772 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-25T14:01:44.835 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:01:44.835 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-25T14:01:44.900 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9 0 2024-04-25T14:01:45.007 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:45.008+0000 7fc041636040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T14:01:45.016 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:45.016+0000 7fde84436040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T14:01:45.047 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.048+0000 7f4fbc034640 1 Processor -- start 2024-04-25T14:01:45.048 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.048+0000 7f4fbc034640 1 -- start start 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fbc034640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fb41576d0 0x7f4fb40d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fbc034640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fbc034640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fb415e3d0 con 0x7f4fb40b36e0 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fbc034640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4fb40b3250 con 0x7f4fb41576d0 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fbc034640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb409bdf0 con 0x7f4fb409b4a0 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fb9dd0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fb41576d0 0x7f4fb40d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fba5d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fb9dd0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fb41576d0 0x7f4fb40d3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52564/0 (socket says 172.21.15.187:52564) 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fba5d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:50472/0 (socket says 172.21.15.187:50472) 2024-04-25T14:01:45.050 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.049+0000 7f4fb9dd0640 1 -- 172.21.15.187:0/645122384 learned_addr learned my addr 172.21.15.187:0/645122384 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fb9dd0640 1 -- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 msgr2=0x7f4fb409b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fb9dd0640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fb9dd0640 1 -- 172.21.15.187:0/645122384 >> v1:172.21.15.150:6789/0 conn(0x7f4fb40b36e0 legacy=0x7f4fb415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fb9dd0640 1 -- 172.21.15.187:0/645122384 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb40b3af0 con 0x7f4fb41576d0 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fba5d1640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:45.051 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.050+0000 7f4fb9dd0640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb41576d0 0x7f4fb40d3320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000bb10 tx=0x7f4fb002c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c8a39e41b961f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fb8dce640 1 -- 172.21.15.187:0/645122384 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0032790 con 0x7f4fb41576d0 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fb8dce640 1 -- 172.21.15.187:0/645122384 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fb0038070 con 0x7f4fb41576d0 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fb8dce640 1 -- 172.21.15.187:0/645122384 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0032b50 con 0x7f4fb41576d0 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fbc034640 1 -- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb41576d0 msgr2=0x7f4fb40d3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fbc034640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb41576d0 0x7f4fb40d3320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000bb10 tx=0x7f4fb002c890 comp rx=0 tx=0).stop 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fbc034640 1 -- 172.21.15.187:0/645122384 shutdown_connections 2024-04-25T14:01:45.052 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fbc034640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb409b4a0 0x7f4fb409b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.053 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.051+0000 7f4fbc034640 1 --2- 172.21.15.187:0/645122384 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb41576d0 0x7f4fb40d3320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.053 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.052+0000 7f4fbc034640 1 -- 172.21.15.187:0/645122384 >> 172.21.15.187:0/645122384 conn(0x7f4fb40b76d0 msgr2=0x7f4fb40b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.053 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.052+0000 7f4fbc034640 1 -- 172.21.15.187:0/645122384 shutdown_connections 2024-04-25T14:01:45.053 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.052+0000 7f4fbc034640 1 -- 172.21.15.187:0/645122384 wait complete. 2024-04-25T14:01:45.053 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:45.054+0000 7fc041636040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T14:01:45.053 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.053+0000 7f4fbc034640 1 Processor -- start 2024-04-25T14:01:45.054 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.053+0000 7f4fbc034640 1 -- start start 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb409b4a0 0x7f4fb414e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40b2ee0 con 0x7f4fb409b4a0 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4fb40b2c00 con 0x7f4fb40b36e0 2024-04-25T14:01:45.056 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fbc034640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fb406e120 con 0x7f4fb41576d0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fb95cf640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:50476/0 (socket says 172.21.15.187:50476) 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 -- 172.21.15.187:0/561555651 learned_addr learned my addr 172.21.15.187:0/561555651 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fb95cf640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52580/0 (socket says 172.21.15.187:52580) 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fb9dd0640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb409b4a0 0x7f4fb414e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 -- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 msgr2=0x7f4fb414cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 -- 172.21.15.187:0/561555651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb409b4a0 msgr2=0x7f4fb414e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb409b4a0 0x7f4fb414e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 -- 172.21.15.187:0/561555651 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fb414e370 con 0x7f4fb41576d0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fb95cf640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4fba5d1640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00efc0 tx=0x7f4fac00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.057 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.055+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac018030 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fac00cdf0 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fb414d5b0 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fac003bb0 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fb414db70 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4fac01b070 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.056+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f4fac01d830 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.057+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fb4076a40 con 0x7f4fb41576d0 2024-04-25T14:01:45.058 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.059+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4fac010040 con 0x7f4fb41576d0 2024-04-25T14:01:45.064 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:45.065+0000 7fde84436040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T14:01:45.131 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.131+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9", "id": 0} v 0) -- 0x7f4fb40b8350 con 0x7f4fb41576d0 2024-04-25T14:01:45.134 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.135+0000 7f4faa7fc640 1 -- 172.21.15.187:0/561555651 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f4fac01db50 con 0x7f4fb41576d0 2024-04-25T14:01:45.134 INFO:teuthology.orchestra.run.smithi187.stdout:0 2024-04-25T14:01:45.135 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.135+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 msgr2=0x7f4fb41f29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.135+0000 7f4fbc034640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4fac00efc0 tx=0x7f4fac00c490 comp rx=0 tx=0).stop 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 shutdown_connections 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fb41576d0 0x7f4fb41f29c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fb40b36e0 0x7f4fb414cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 --2- 172.21.15.187:0/561555651 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fb409b4a0 0x7f4fb414e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 >> 172.21.15.187:0/561555651 conn(0x7f4fb40b76d0 msgr2=0x7f4fb415fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 shutdown_connections 2024-04-25T14:01:45.136 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.136+0000 7f4fbc034640 1 -- 172.21.15.187:0/561555651 wait complete. 2024-04-25T14:01:45.146 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 377acd06-0a97-46c7-9f7e-904629dd9b46 1 2024-04-25T14:01:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.254+0000 7ffa58653640 1 Processor -- start 2024-04-25T14:01:45.254 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.254+0000 7ffa58653640 1 -- start start 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa58653640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffa5015df70 0x7ffa500d1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa58653640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa500d2a40 con 0x7ffa500d2530 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa58653640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa500d2d20 con 0x7ffa500733e0 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa58653640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ffa500d3000 con 0x7ffa5015df70 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa56bf0640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffa500733e0 0x7ffa500b6fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44706/0 (socket says 172.21.15.187:44706) 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.255+0000 7ffa56bf0640 1 -- 172.21.15.187:0/84145596 learned_addr learned my addr 172.21.15.187:0/84145596 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa563ef640 1 --2- 172.21.15.187:0/84145596 >> v2:172.21.15.150:3300/0 conn(0x7ffa5015df70 0x7ffa500d1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3700442701 0 0) 0x7ffa500d2d20 con 0x7ffa500733e0 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa3c003520 con 0x7ffa500733e0 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa563ef640 1 -- 172.21.15.187:0/84145596 >> v1:172.21.15.187:6789/0 conn(0x7ffa500733e0 legacy=0x7ffa500b6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa563ef640 1 -- 172.21.15.187:0/84145596 >> v1:172.21.15.150:6789/0 conn(0x7ffa500d2530 legacy=0x7ffa50072cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783505236 0 0) 0x7ffa500d2a40 con 0x7ffa500d2530 2024-04-25T14:01:45.256 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.256+0000 7ffa563ef640 1 -- 172.21.15.187:0/84145596 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa500d32e0 con 0x7ffa5015df70 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.257+0000 7ffa563ef640 1 --2- 172.21.15.187:0/84145596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa5015df70 0x7ffa500d1ff0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0097a0 tx=0x7ffa4c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56cf48b0aa1ef5cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.257+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c037070 con 0x7ffa5015df70 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.257+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c032440 con 0x7ffa5015df70 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.257+0000 7ffa553ed640 1 -- 172.21.15.187:0/84145596 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c0327f0 con 0x7ffa5015df70 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 -- 172.21.15.187:0/84145596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa5015df70 msgr2=0x7ffa500d1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 --2- 172.21.15.187:0/84145596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa5015df70 0x7ffa500d1ff0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c0097a0 tx=0x7ffa4c02a230 comp rx=0 tx=0).stop 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 -- 172.21.15.187:0/84145596 shutdown_connections 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 --2- 172.21.15.187:0/84145596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa5015df70 0x7ffa500d1ff0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 -- 172.21.15.187:0/84145596 >> 172.21.15.187:0/84145596 conn(0x7ffa50071530 msgr2=0x7ffa50071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 -- 172.21.15.187:0/84145596 shutdown_connections 2024-04-25T14:01:45.258 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.258+0000 7ffa58653640 1 -- 172.21.15.187:0/84145596 wait complete. 2024-04-25T14:01:45.259 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.260+0000 7ffa58653640 1 Processor -- start 2024-04-25T14:01:45.259 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.260+0000 7ffa58653640 1 -- start start 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 0x7ffa50157cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa500c8f10 0x7ffa500c9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffa500d0270 con 0x7ffa500d2530 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5015e380 con 0x7ffa500c8f10 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa58653640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffa500d03f0 con 0x7ffa5015df70 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa55bee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 0x7ffa50157cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.187:48706/0 (socket says 172.21.15.187:48706) 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa55bee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 0x7ffa50157cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:50486/0 (socket says 172.21.15.187:50486) 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 -- 172.21.15.187:0/3003999970 learned_addr learned my addr 172.21.15.187:0/3003999970 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa56bf0640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa500c8f10 0x7ffa500c9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.263 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 -- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 msgr2=0x7ffa50157cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 0x7ffa50157cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 -- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa500c8f10 msgr2=0x7ffa500c9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa500c8f10 0x7ffa500c9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.261+0000 7ffa563ef640 1 -- 172.21.15.187:0/3003999970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa501f47f0 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa563ef640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02a270 tx=0x7ffa4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c037040 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c042070 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa501f3c70 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c030900 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa501f4230 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ffa4c030aa0 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.262+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7ffa4c047cb0 con 0x7ffa500d2530 2024-04-25T14:01:45.264 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.263+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa50076a40 con 0x7ffa500d2530 2024-04-25T14:01:45.265 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.265+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ffa4c047460 con 0x7ffa500d2530 2024-04-25T14:01:45.337 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.337+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd new", "uuid": "377acd06-0a97-46c7-9f7e-904629dd9b46", "id": 1} v 0) -- 0x7ffa500b2db0 con 0x7ffa500d2530 2024-04-25T14:01:45.341 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.341+0000 7ffa46ffd640 1 -- 172.21.15.187:0/3003999970 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "377acd06-0a97-46c7-9f7e-904629dd9b46", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7ffa4c047640 con 0x7ffa500d2530 2024-04-25T14:01:45.341 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-25T14:01:45.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 msgr2=0x7ffa500c7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.342 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c02a270 tx=0x7ffa4c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 shutdown_connections 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffa500c8f10 0x7ffa500c9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffa5015df70 0x7ffa50157cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 --2- 172.21.15.187:0/3003999970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffa500d2530 0x7ffa500c7460 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 >> 172.21.15.187:0/3003999970 conn(0x7ffa50071530 msgr2=0x7ffa500b4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 shutdown_connections 2024-04-25T14:01:45.343 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.342+0000 7ffa58653640 1 -- 172.21.15.187:0/3003999970 wait complete. 2024-04-25T14:01:45.353 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 09bfcf89-4e73-43f8-9276-41f57000458f 2 2024-04-25T14:01:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.462+0000 7f4827bee640 1 Processor -- start 2024-04-25T14:01:45.462 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.463+0000 7f4827bee640 1 -- start start 2024-04-25T14:01:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f4827bee640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48201576d0 0x7f48200d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f4827bee640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f482015e3d0 con 0x7f48200b36e0 2024-04-25T14:01:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f4827bee640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48200b3af0 con 0x7f482009b4a0 2024-04-25T14:01:45.464 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f4827bee640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f48200b3250 con 0x7f48201576d0 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48201576d0 0x7f48200d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f48201576d0 0x7f48200d3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52600/0 (socket says 172.21.15.187:52600) 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482618b640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f482009b4a0 0x7f482009b8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44710/0 (socket says 172.21.15.187:44710) 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 -- 172.21.15.187:0/4062893555 learned_addr learned my addr 172.21.15.187:0/4062893555 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 -- 172.21.15.187:0/4062893555 >> v1:172.21.15.187:6789/0 conn(0x7f482009b4a0 legacy=0x7f482009b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 -- 172.21.15.187:0/4062893555 >> v1:172.21.15.150:6789/0 conn(0x7f48200b36e0 legacy=0x7f482015db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.465 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.464+0000 7f482598a640 1 -- 172.21.15.187:0/4062893555 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48200b2920 con 0x7f48201576d0 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.465+0000 7f482598a640 1 --2- 172.21.15.187:0/4062893555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48201576d0 0x7f48200d3320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4814009870 tx=0x7f481402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3ac7cd3d3bfa8d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.465+0000 7f4824988640 1 -- 172.21.15.187:0/4062893555 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814037070 con 0x7f48201576d0 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.465+0000 7f4824988640 1 -- 172.21.15.187:0/4062893555 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4814032440 con 0x7f48201576d0 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.465+0000 7f4824988640 1 -- 172.21.15.187:0/4062893555 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4814032810 con 0x7f48201576d0 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 -- 172.21.15.187:0/4062893555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48201576d0 msgr2=0x7f48200d3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.466 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 --2- 172.21.15.187:0/4062893555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48201576d0 0x7f48200d3320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4814009870 tx=0x7f481402a270 comp rx=0 tx=0).stop 2024-04-25T14:01:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 -- 172.21.15.187:0/4062893555 shutdown_connections 2024-04-25T14:01:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 --2- 172.21.15.187:0/4062893555 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48201576d0 0x7f48200d3320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 -- 172.21.15.187:0/4062893555 >> 172.21.15.187:0/4062893555 conn(0x7f48200b76d0 msgr2=0x7f48200b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 -- 172.21.15.187:0/4062893555 shutdown_connections 2024-04-25T14:01:45.467 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.466+0000 7f4827bee640 1 -- 172.21.15.187:0/4062893555 wait complete. 2024-04-25T14:01:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.468+0000 7f4827bee640 1 Processor -- start 2024-04-25T14:01:45.468 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.468+0000 7f4827bee640 1 -- start start 2024-04-25T14:01:45.470 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 0x7f48201f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.470 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 0x7f48201f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.470 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48201576d0 0x7f48201585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f482006e120 con 0x7f48201576d0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f48200b2ee0 con 0x7f48200b36e0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4827bee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f482006e2a0 con 0x7f482009b4a0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f482598a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 0x7f48201f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f4825189640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 0x7f48201f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f482598a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 0x7f48201f6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:50488/0 (socket says 172.21.15.187:50488) 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.469+0000 7f482598a640 1 -- 172.21.15.187:0/3873739157 learned_addr learned my addr 172.21.15.187:0/3873739157 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f482618b640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48201576d0 0x7f48201585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 -- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 msgr2=0x7f48201f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 0x7f48201f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 -- 172.21.15.187:0/3873739157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48201576d0 msgr2=0x7f48201585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48201576d0 0x7f48201585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 -- 172.21.15.187:0/3873739157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4820157d20 con 0x7f48200b36e0 2024-04-25T14:01:45.471 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4825189640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 0x7f48201f2a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c3c0 tx=0x7f481c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c0052d0 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f481c005490 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48201f56a0 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f481c007260 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48200d0f70 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f481c007400 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.470+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f481c011590 con 0x7f48200b36e0 2024-04-25T14:01:45.472 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.471+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4820076a40 con 0x7f48200b36e0 2024-04-25T14:01:45.473 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.473+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f481c0035f0 con 0x7f48200b36e0 2024-04-25T14:01:45.545 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.546+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "09bfcf89-4e73-43f8-9276-41f57000458f", "id": 2} v 0) -- 0x7f48200b25d0 con 0x7f48200b36e0 2024-04-25T14:01:45.551 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f48127fc640 1 -- 172.21.15.187:0/3873739157 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "09bfcf89-4e73-43f8-9276-41f57000458f", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f481c003b80 con 0x7f48200b36e0 2024-04-25T14:01:45.551 INFO:teuthology.orchestra.run.smithi187.stdout:2 2024-04-25T14:01:45.552 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 msgr2=0x7f48201f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 0x7f48201f2a90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f481c00c3c0 tx=0x7f481c00c880 comp rx=0 tx=0).stop 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 shutdown_connections 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f48201576d0 0x7f48201585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f48200b36e0 0x7f48201f2a90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 --2- 172.21.15.187:0/3873739157 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f482009b4a0 0x7f48201f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.552+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 >> 172.21.15.187:0/3873739157 conn(0x7f48200b76d0 msgr2=0x7f482006e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.553+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 shutdown_connections 2024-04-25T14:01:45.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.553+0000 7f4827bee640 1 -- 172.21.15.187:0/3873739157 wait complete. 2024-04-25T14:01:45.563 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 34d07c8c-b115-4f61-a34d-e6133022dc1e 3 2024-04-25T14:01:45.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.671+0000 7fee79706640 1 Processor -- start 2024-04-25T14:01:45.671 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.671+0000 7fee79706640 1 -- start start 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee79706640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee74160150 0x7fee740b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee79706640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee7406d800 0x7fee740d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee79706640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee740d4f20 con 0x7fee740b4c10 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee79706640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fee7406cc60 con 0x7fee74160150 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee79706640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fee74160560 con 0x7fee7406d800 2024-04-25T14:01:45.673 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee737fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee7406d800 0x7fee740d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee72ffd640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fee740b4c10 0x7fee7406d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44720/0 (socket says 172.21.15.187:44720) 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee72ffd640 1 -- 172.21.15.187:0/3945519734 learned_addr learned my addr 172.21.15.187:0/3945519734 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee727fc640 1 --2- 172.21.15.187:0/3945519734 >> v2:172.21.15.150:3300/0 conn(0x7fee74160150 0x7fee740b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee71ffb640 1 -- 172.21.15.187:0/3945519734 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3609490338 0 0) 0x7fee740d4f20 con 0x7fee740b4c10 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee71ffb640 1 -- 172.21.15.187:0/3945519734 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee60003520 con 0x7fee740b4c10 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee737fe640 1 -- 172.21.15.187:0/3945519734 >> v2:172.21.15.150:3300/0 conn(0x7fee74160150 msgr2=0x7fee740b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee737fe640 1 --2- 172.21.15.187:0/3945519734 >> v2:172.21.15.150:3300/0 conn(0x7fee74160150 0x7fee740b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee737fe640 1 -- 172.21.15.187:0/3945519734 >> v1:172.21.15.187:6789/0 conn(0x7fee740b4c10 legacy=0x7fee7406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.674 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.673+0000 7fee737fe640 1 -- 172.21.15.187:0/3945519734 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee740d5200 con 0x7fee7406d800 2024-04-25T14:01:45.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.674+0000 7fee737fe640 1 --2- 172.21.15.187:0/3945519734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee7406d800 0x7fee740d0950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fee740d0690 tx=0x7fee6c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f7d4781b5d0a0b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.675 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.674+0000 7fee71ffb640 1 -- 172.21.15.187:0/3945519734 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c007740 con 0x7fee7406d800 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.674+0000 7fee71ffb640 1 -- 172.21.15.187:0/3945519734 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6c0078e0 con 0x7fee7406d800 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.674+0000 7fee71ffb640 1 -- 172.21.15.187:0/3945519734 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6c007cb0 con 0x7fee7406d800 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 -- 172.21.15.187:0/3945519734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee7406d800 msgr2=0x7fee740d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 --2- 172.21.15.187:0/3945519734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee7406d800 0x7fee740d0950 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fee740d0690 tx=0x7fee6c02a680 comp rx=0 tx=0).stop 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 -- 172.21.15.187:0/3945519734 shutdown_connections 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 --2- 172.21.15.187:0/3945519734 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee7406d800 0x7fee740d0950 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 --2- 172.21.15.187:0/3945519734 >> v2:172.21.15.150:3300/0 conn(0x7fee74160150 0x7fee740b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 -- 172.21.15.187:0/3945519734 >> 172.21.15.187:0/3945519734 conn(0x7fee740b7690 msgr2=0x7fee740b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 -- 172.21.15.187:0/3945519734 shutdown_connections 2024-04-25T14:01:45.676 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.675+0000 7fee79706640 1 -- 172.21.15.187:0/3945519734 wait complete. 2024-04-25T14:01:45.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.677+0000 7fee79706640 1 Processor -- start 2024-04-25T14:01:45.677 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.677+0000 7fee79706640 1 -- start start 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7406d800 0x7fee7414abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 0x7fee7414ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee7406c8f0 con 0x7fee7406d800 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee7406c610 con 0x7fee740b4c10 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee79706640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee7406dc10 con 0x7fee74160150 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee727fc640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee727fc640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52640/0 (socket says 172.21.15.187:52640) 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee737fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 0x7fee7414ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.680 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee727fc640 1 -- 172.21.15.187:0/3321501240 learned_addr learned my addr 172.21.15.187:0/3321501240 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.678+0000 7fee72ffd640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7406d800 0x7fee7414abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 -- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 msgr2=0x7fee7414ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 0x7fee7414ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 -- 172.21.15.187:0/3321501240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7406d800 msgr2=0x7fee7414abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7406d800 0x7fee7414abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 -- 172.21.15.187:0/3321501240 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee7414b0f0 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee737fe640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 0x7fee7414ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee727fc640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00e8a0 tx=0x7fee5c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c017070 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee5c00c940 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7414b420 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee5c00cd10 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7414b9e0 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fee5c01a070 con 0x7fee740b4c10 2024-04-25T14:01:45.681 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.679+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fee5c013470 con 0x7fee740b4c10 2024-04-25T14:01:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.680+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee74076a40 con 0x7fee740b4c10 2024-04-25T14:01:45.682 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.682+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fee5c006020 con 0x7fee740b4c10 2024-04-25T14:01:45.755 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.755+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "34d07c8c-b115-4f61-a34d-e6133022dc1e", "id": 3} v 0) -- 0x7fee740d4030 con 0x7fee740b4c10 2024-04-25T14:01:45.760 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.761+0000 7fee6b7fe640 1 -- 172.21.15.187:0/3321501240 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "34d07c8c-b115-4f61-a34d-e6133022dc1e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fee5c006480 con 0x7fee740b4c10 2024-04-25T14:01:45.761 INFO:teuthology.orchestra.run.smithi187.stdout:3 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 msgr2=0x7fee7414c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fee5c00e8a0 tx=0x7fee5c00ed60 comp rx=0 tx=0).stop 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 shutdown_connections 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee74160150 0x7fee7414ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee740b4c10 0x7fee7414c4c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 --2- 172.21.15.187:0/3321501240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee7406d800 0x7fee7414abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 >> 172.21.15.187:0/3321501240 conn(0x7fee740b7690 msgr2=0x7fee740b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 shutdown_connections 2024-04-25T14:01:45.762 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.762+0000 7fee79706640 1 -- 172.21.15.187:0/3321501240 wait complete. 2024-04-25T14:01:45.770 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:45.771+0000 7fc041636040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T14:01:45.773 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 17dd1c65-44ee-44c4-9f4d-00a5884fb4dd 4 2024-04-25T14:01:45.837 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:45.837+0000 7fde84436040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T14:01:45.858 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:45.859+0000 7fc041636040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T14:01:45.882 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.882+0000 7fae2e4bd640 1 Processor -- start 2024-04-25T14:01:45.882 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.882+0000 7fae2e4bd640 1 -- start start 2024-04-25T14:01:45.883 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2e4bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.883 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2e4bd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae280d2dc0 0x7fae281604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.883 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2e4bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae280b9860 con 0x7fae280b67b0 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2e4bd640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fae280b92a0 con 0x7fae280d2dc0 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2e4bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae280b9580 con 0x7fae28164780 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae277fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2ca5a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae280d2dc0 0x7fae281604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae277fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.187:50506/0 (socket says 172.21.15.187:50506) 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae2ca5a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fae280d2dc0 0x7fae281604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52648/0 (socket says 172.21.15.187:52648) 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.883+0000 7fae277fe640 1 -- 172.21.15.187:0/3506509926 learned_addr learned my addr 172.21.15.187:0/3506509926 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae2ca5a640 1 -- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 msgr2=0x7fae280b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae2ca5a640 1 -- 172.21.15.187:0/3506509926 >> v1:172.21.15.150:6789/0 conn(0x7fae280b67b0 legacy=0x7fae28164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae2ca5a640 1 -- 172.21.15.187:0/3506509926 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae280d15d0 con 0x7fae280d2dc0 2024-04-25T14:01:45.884 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae277fe640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:45.885 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281604c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae20009d90 tx=0x7fae2002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bf80bbd03b68091 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae26ffd640 1 -- 172.21.15.187:0/3506509926 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20031600 con 0x7fae280d2dc0 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae26ffd640 1 -- 172.21.15.187:0/3506509926 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae200317a0 con 0x7fae280d2dc0 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.884+0000 7fae26ffd640 1 -- 172.21.15.187:0/3506509926 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20031b70 con 0x7fae280d2dc0 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 -- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 msgr2=0x7fae281604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281604c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fae20009d90 tx=0x7fae2002a5d0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 -- 172.21.15.187:0/3506509926 shutdown_connections 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281604c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/3506509926 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae28164780 0x7fae280b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 -- 172.21.15.187:0/3506509926 >> 172.21.15.187:0/3506509926 conn(0x7fae280d3960 msgr2=0x7fae280d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 -- 172.21.15.187:0/3506509926 shutdown_connections 2024-04-25T14:01:45.886 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.885+0000 7fae2e4bd640 1 -- 172.21.15.187:0/3506509926 wait complete. 2024-04-25T14:01:45.887 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.887+0000 7fae2e4bd640 1 Processor -- start 2024-04-25T14:01:45.887 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.887+0000 7fae2e4bd640 1 -- start start 2024-04-25T14:01:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.888+0000 7fae2e4bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae280b67b0 0x7fae28155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 0x7fae280cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fae280b8fe0 con 0x7fae280b67b0 2024-04-25T14:01:45.890 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fae280b8e60 con 0x7fae280d2dc0 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fae28160a00 con 0x7fae281eab30 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae277fe640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae277fe640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52656/0 (socket says 172.21.15.187:52656) 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae277fe640 1 -- 172.21.15.187:0/1663502260 learned_addr learned my addr 172.21.15.187:0/1663502260 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 0x7fae280cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae27fff640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae280b67b0 0x7fae28155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 -- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 msgr2=0x7fae281ea5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 -- 172.21.15.187:0/1663502260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae280b67b0 msgr2=0x7fae28155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae280b67b0 0x7fae28155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 -- 172.21.15.187:0/1663502260 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2815eb00 con 0x7fae281eab30 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae277fe640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2ca5a640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 0x7fae280cf3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae20031000 tx=0x7fae20004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:45.891 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2003f070 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae20030b60 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae2815da60 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae20009450 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.889+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae2815e020 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.890+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fae20044460 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.890+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fae2003b920 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.890+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae280b2ac0 con 0x7fae281eab30 2024-04-25T14:01:45.892 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.892+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fae280b2ac0 con 0x7fae281eab30 2024-04-25T14:01:45.902 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:45.904+0000 7fc041636040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T14:01:45.930 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:45.931+0000 7fde84436040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T14:01:45.965 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.965+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd new", "uuid": "17dd1c65-44ee-44c4-9f4d-00a5884fb4dd", "id": 4} v 0) -- 0x7fae28076a40 con 0x7fae281eab30 2024-04-25T14:01:45.970 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.970+0000 7fae24ff9640 1 -- 172.21.15.187:0/1663502260 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "17dd1c65-44ee-44c4-9f4d-00a5884fb4dd", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fae20052020 con 0x7fae281eab30 2024-04-25T14:01:45.970 INFO:teuthology.orchestra.run.smithi187.stdout:4 2024-04-25T14:01:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 msgr2=0x7fae280cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:45.971 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 0x7fae280cf3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fae20031000 tx=0x7fae20004a40 comp rx=0 tx=0).stop 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 shutdown_connections 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fae281eab30 0x7fae280cf3f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fae280d2dc0 0x7fae281ea5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 --2- 172.21.15.187:0/1663502260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fae280b67b0 0x7fae28155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.971+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 >> 172.21.15.187:0/1663502260 conn(0x7fae280d3960 msgr2=0x7fae280b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.972+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 shutdown_connections 2024-04-25T14:01:45.972 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:45.972+0000 7fae2e4bd640 1 -- 172.21.15.187:0/1663502260 wait complete. 2024-04-25T14:01:45.975 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:45.975+0000 7fde84436040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T14:01:45.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 19c9fb99-f307-4eab-99bf-1c6cbafc1148 5 2024-04-25T14:01:46.070 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.071+0000 7fc041636040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T14:01:46.090 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.090+0000 7f138c29f640 1 Processor -- start 2024-04-25T14:01:46.090 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.090+0000 7f138c29f640 1 -- start start 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.091+0000 7f138c29f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13840b4110 0x7f13841605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.091+0000 7f138c29f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13840d3570 con 0x7f13840cf8e0 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.091+0000 7f138c29f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13840b4800 con 0x7f13840cdfe0 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.091+0000 7f138c29f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f13840b4ae0 con 0x7f13840b4110 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138983a640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f13840cdfe0 0x7f13840d1ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44722/0 (socket says 172.21.15.187:44722) 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138983a640 1 -- 172.21.15.187:0/1891692618 learned_addr learned my addr 172.21.15.187:0/1891692618 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138a83c640 1 --2- 172.21.15.187:0/1891692618 >> v2:172.21.15.150:3300/0 conn(0x7f13840b4110 0x7f13841605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 712797077 0 0) 0x7f13840b4800 con 0x7f13840cdfe0 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1370003520 con 0x7f13840cdfe0 2024-04-25T14:01:46.092 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3511751037 0 0) 0x7f13840d3570 con 0x7f13840cf8e0 2024-04-25T14:01:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13840b4800 con 0x7f13840cf8e0 2024-04-25T14:01:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138a83c640 1 -- 172.21.15.187:0/1891692618 >> v1:172.21.15.187:6789/0 conn(0x7f13840cdfe0 legacy=0x7f13840d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138a83c640 1 -- 172.21.15.187:0/1891692618 >> v1:172.21.15.150:6789/0 conn(0x7f13840cf8e0 legacy=0x7f13840b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.093 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.092+0000 7f138a83c640 1 -- 172.21.15.187:0/1891692618 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13840b4dc0 con 0x7f13840b4110 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.093+0000 7f138a83c640 1 --2- 172.21.15.187:0/1891692618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840b4110 0x7f13841605d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f137c009d90 tx=0x7f137c02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bd7e56c06e68b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.093+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c007740 con 0x7f13840b4110 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.093+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c0078e0 con 0x7f13840b4110 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.093+0000 7f1389039640 1 -- 172.21.15.187:0/1891692618 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c007cb0 con 0x7f13840b4110 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 -- 172.21.15.187:0/1891692618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840b4110 msgr2=0x7f13841605d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 --2- 172.21.15.187:0/1891692618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840b4110 0x7f13841605d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f137c009d90 tx=0x7f137c02a6f0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.094 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 -- 172.21.15.187:0/1891692618 shutdown_connections 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 --2- 172.21.15.187:0/1891692618 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840b4110 0x7f13841605d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 -- 172.21.15.187:0/1891692618 >> 172.21.15.187:0/1891692618 conn(0x7f138409b4a0 msgr2=0x7f138409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 -- 172.21.15.187:0/1891692618 shutdown_connections 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.094+0000 7f138c29f640 1 -- 172.21.15.187:0/1891692618 wait complete. 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.096+0000 7f138c29f640 1 Processor -- start 2024-04-25T14:01:46.095 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.096+0000 7f138c29f640 1 -- start start 2024-04-25T14:01:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 0x7f1384156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840cf8e0 0x7f138415cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f138416b320 con 0x7f13840cf8e0 2024-04-25T14:01:46.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f138416b1a0 con 0x7f13840b4110 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138c29f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f138416b4a0 con 0x7f13840cdfe0 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138983a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 0x7f1384156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52672/0 (socket says 172.21.15.187:52672) 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 -- 172.21.15.187:0/3926928665 learned_addr learned my addr 172.21.15.187:0/3926928665 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a83c640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840cf8e0 0x7f138415cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 -- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 msgr2=0x7f1384156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 0x7f1384156e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 -- 172.21.15.187:0/3926928665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840cf8e0 msgr2=0x7f138415cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840cf8e0 0x7f138415cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 -- 172.21.15.187:0/3926928665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f138415d4a0 con 0x7f13840b4110 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138983a640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 0x7f1384156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.097+0000 7f138a03b640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f137400c430 tx=0x7f137400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.099 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374017070 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13740052f0 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f138415bec0 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374003830 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f138415c480 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f137401a070 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.098+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f1374011bd0 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.099+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138415fa70 con 0x7f13840b4110 2024-04-25T14:01:46.100 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.101+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1374021020 con 0x7f13840b4110 2024-04-25T14:01:46.119 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.119+0000 7fc041636040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T14:01:46.152 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:46.152+0000 7fde84436040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T14:01:46.171 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.172+0000 7fc041636040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T14:01:46.172 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.173+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "19c9fb99-f307-4eab-99bf-1c6cbafc1148", "id": 5} v 0) -- 0x7f1384076a40 con 0x7f13840b4110 2024-04-25T14:01:46.178 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.179+0000 7f137affd640 1 -- 172.21.15.187:0/3926928665 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "19c9fb99-f307-4eab-99bf-1c6cbafc1148", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f1374011590 con 0x7f13840b4110 2024-04-25T14:01:46.179 INFO:teuthology.orchestra.run.smithi187.stdout:5 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 msgr2=0x7f13840b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f137400c430 tx=0x7f137400c8f0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 shutdown_connections 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13840cf8e0 0x7f138415cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13840cdfe0 0x7f1384156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 --2- 172.21.15.187:0/3926928665 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13840b4110 0x7f13840b3120 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 >> 172.21.15.187:0/3926928665 conn(0x7f138409b4a0 msgr2=0x7f138415f360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.180 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 shutdown_connections 2024-04-25T14:01:46.181 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.180+0000 7f138c29f640 1 -- 172.21.15.187:0/3926928665 wait complete. 2024-04-25T14:01:46.191 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new c125fda9-d826-4710-a37d-d5d2f8b67818 6 2024-04-25T14:01:46.202 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:46.203+0000 7fde84436040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T14:01:46.259 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:46.259+0000 7fde84436040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T14:01:46.284 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.285+0000 7fc041636040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T14:01:46.299 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.300+0000 7f67bee84640 1 Processor -- start 2024-04-25T14:01:46.299 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.300+0000 7f67bee84640 1 -- start start 2024-04-25T14:01:46.301 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bee84640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67b80b30b0 0x7f67b80cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.301 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bee84640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67b8160720 con 0x7f67b80cf9c0 2024-04-25T14:01:46.301 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bee84640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67b80b51a0 con 0x7f67b80ceae0 2024-04-25T14:01:46.301 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bee84640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f67b80cfdd0 con 0x7f67b80b30b0 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bcc20640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f67b80b30b0 0x7f67b80cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67b7fff640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f67b80ceae0 0x7f67b80b2030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:44736/0 (socket says 172.21.15.187:44736) 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67b7fff640 1 -- 172.21.15.187:0/1960670963 learned_addr learned my addr 172.21.15.187:0/1960670963 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bcc20640 1 -- 172.21.15.187:0/1960670963 >> v1:172.21.15.187:6789/0 conn(0x7f67b80ceae0 legacy=0x7f67b80b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67bcc20640 1 -- 172.21.15.187:0/1960670963 >> v1:172.21.15.150:6789/0 conn(0x7f67b80cf9c0 legacy=0x7f67b80b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.301+0000 7f67b77fe640 1 -- 172.21.15.187:0/1960670963 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416243983 0 0) 0x7f67b8160720 con 0x7f67b80cf9c0 2024-04-25T14:01:46.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.302+0000 7f67bcc20640 1 -- 172.21.15.187:0/1960670963 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80b2e50 con 0x7f67b80b30b0 2024-04-25T14:01:46.303 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.302+0000 7f67bcc20640 1 --2- 172.21.15.187:0/1960670963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cf480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009720 tx=0x7f67ac02a150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59d06a211c8f34e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.303 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.302+0000 7f67b77fe640 1 -- 172.21.15.187:0/1960670963 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac037070 con 0x7f67b80b30b0 2024-04-25T14:01:46.303 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.302+0000 7f67b77fe640 1 -- 172.21.15.187:0/1960670963 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac032440 con 0x7f67b80b30b0 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.302+0000 7f67b77fe640 1 -- 172.21.15.187:0/1960670963 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac02ae60 con 0x7f67b80b30b0 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 -- 172.21.15.187:0/1960670963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 msgr2=0x7f67b80cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 --2- 172.21.15.187:0/1960670963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cf480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f67ac009720 tx=0x7f67ac02a150 comp rx=0 tx=0).stop 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 -- 172.21.15.187:0/1960670963 shutdown_connections 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 --2- 172.21.15.187:0/1960670963 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cf480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 -- 172.21.15.187:0/1960670963 >> 172.21.15.187:0/1960670963 conn(0x7f67b809b4a0 msgr2=0x7f67b809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 -- 172.21.15.187:0/1960670963 shutdown_connections 2024-04-25T14:01:46.304 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.303+0000 7f67bee84640 1 -- 172.21.15.187:0/1960670963 wait complete. 2024-04-25T14:01:46.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.305+0000 7f67bee84640 1 Processor -- start 2024-04-25T14:01:46.305 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.305+0000 7f67bee84640 1 -- start start 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b80ceae0 0x7f67b80c8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 0x7f67b806de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f67b80d3070 con 0x7f67b80ceae0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f67b80d2d90 con 0x7f67b80b30b0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.306+0000 7f67bee84640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f67b80b3c70 con 0x7f67b80cf9c0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:52686/0 (socket says 172.21.15.187:52686) 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 -- 172.21.15.187:0/2491097893 learned_addr learned my addr 172.21.15.187:0/2491097893 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bd421640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 0x7f67b806de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67b7fff640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b80ceae0 0x7f67b80c8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.308 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 -- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 msgr2=0x7f67b806de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 0x7f67b806de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 -- 172.21.15.187:0/2491097893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b80ceae0 msgr2=0x7f67b80c8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b80ceae0 0x7f67b80c8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 -- 172.21.15.187:0/2491097893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67b80c7020 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bd421640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 0x7f67b806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bcc20640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f67ac035040 tx=0x7f67ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac037040 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ac004230 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67b81f9dc0 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ac03c670 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.307+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67b81fa380 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.308+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f67ac03a070 con 0x7f67b80b30b0 2024-04-25T14:01:46.309 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.308+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f67ac003de0 con 0x7f67b80b30b0 2024-04-25T14:01:46.310 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.309+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67b80c71f0 con 0x7f67b80b30b0 2024-04-25T14:01:46.310 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.311+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f67ac041020 con 0x7f67b80b30b0 2024-04-25T14:01:46.378 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:46.378+0000 7fde84436040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T14:01:46.383 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.384+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c125fda9-d826-4710-a37d-d5d2f8b67818", "id": 6} v 0) -- 0x7f67b8076a40 con 0x7f67b80b30b0 2024-04-25T14:01:46.389 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.389+0000 7f67b57fa640 1 -- 172.21.15.187:0/2491097893 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c125fda9-d826-4710-a37d-d5d2f8b67818", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f67ac0037f0 con 0x7f67b80b30b0 2024-04-25T14:01:46.389 INFO:teuthology.orchestra.run.smithi187.stdout:6 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 msgr2=0x7f67b80cb870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f67ac035040 tx=0x7f67ac0046c0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 shutdown_connections 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f67b80cf9c0 0x7f67b806de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f67b80ceae0 0x7f67b80c8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 --2- 172.21.15.187:0/2491097893 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f67b80b30b0 0x7f67b80cb870 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.391 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 >> 172.21.15.187:0/2491097893 conn(0x7f67b809b4a0 msgr2=0x7f67b80cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.392 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 shutdown_connections 2024-04-25T14:01:46.392 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.391+0000 7f67bee84640 1 -- 172.21.15.187:0/2491097893 wait complete. 2024-04-25T14:01:46.405 DEBUG:teuthology.orchestra.run.smithi187:> sudo ceph --cluster ceph osd new 98942485-6160-4b36-8909-62f908571405 7 2024-04-25T14:01:46.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.554+0000 7f3999ba8640 1 Processor -- start 2024-04-25T14:01:46.553 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.554+0000 7f3999ba8640 1 -- start start 2024-04-25T14:01:46.555 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3999ba8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f39940cdac0 0x7f399406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.555 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3999ba8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f399406cab0 0x7f39940b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.555 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3999ba8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39940ce0d0 con 0x7f399415e240 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3999ba8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f39940cf840 con 0x7f39940cdac0 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3999ba8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f39940ce370 con 0x7f399406cab0 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3993fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f399406cab0 0x7f39940b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f39937fe640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f399415e240 0x7f39940cf130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.187:41792/0 (socket says 172.21.15.187:41792) 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f39937fe640 1 -- 172.21.15.187:0/2257438528 learned_addr learned my addr 172.21.15.187:0/2257438528 (peer_addr_for_me v1:172.21.15.187:0/0) 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3993fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f399406cab0 0x7f39940b30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:60604/0 (socket says 172.21.15.187:60604) 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3992ffd640 1 --2- 172.21.15.187:0/2257438528 >> v2:172.21.15.150:3300/0 conn(0x7f39940cdac0 0x7f399406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3993fff640 1 -- 172.21.15.187:0/2257438528 >> v2:172.21.15.150:3300/0 conn(0x7f39940cdac0 msgr2=0x7f399406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3993fff640 1 --2- 172.21.15.187:0/2257438528 >> v2:172.21.15.150:3300/0 conn(0x7f39940cdac0 0x7f399406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f3993fff640 1 -- 172.21.15.187:0/2257438528 >> v1:172.21.15.187:6789/0 conn(0x7f399415e240 legacy=0x7f39940cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.556+0000 7f3993fff640 1 -- 172.21.15.187:0/2257438528 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39940ce650 con 0x7f399406cab0 2024-04-25T14:01:46.556 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.555+0000 7f39927fc640 1 -- 172.21.15.187:0/2257438528 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149986831 0 0) 0x7f39940ce0d0 con 0x7f399415e240 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.556+0000 7f3993fff640 1 --2- 172.21.15.187:0/2257438528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399406cab0 0x7f39940b30a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f39940ce2a0 tx=0x7f398c02a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=369437df1f6ffebe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.556+0000 7f39927fc640 1 -- 172.21.15.187:0/2257438528 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c007740 con 0x7f399406cab0 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.556+0000 7f39927fc640 1 -- 172.21.15.187:0/2257438528 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398c0078e0 con 0x7f399406cab0 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.556+0000 7f39927fc640 1 -- 172.21.15.187:0/2257438528 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c007cb0 con 0x7f399406cab0 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 -- 172.21.15.187:0/2257438528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399406cab0 msgr2=0x7f39940b30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 --2- 172.21.15.187:0/2257438528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399406cab0 0x7f39940b30a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f39940ce2a0 tx=0x7f398c02a540 comp rx=0 tx=0).stop 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 -- 172.21.15.187:0/2257438528 shutdown_connections 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 --2- 172.21.15.187:0/2257438528 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399406cab0 0x7f39940b30a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 --2- 172.21.15.187:0/2257438528 >> v2:172.21.15.150:3300/0 conn(0x7f39940cdac0 0x7f399406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 -- 172.21.15.187:0/2257438528 >> 172.21.15.187:0/2257438528 conn(0x7f399409b4a0 msgr2=0x7f39940b6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 -- 172.21.15.187:0/2257438528 shutdown_connections 2024-04-25T14:01:46.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.557+0000 7f3999ba8640 1 -- 172.21.15.187:0/2257438528 wait complete. 2024-04-25T14:01:46.559 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.559+0000 7f3999ba8640 1 Processor -- start 2024-04-25T14:01:46.559 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.559+0000 7f3999ba8640 1 -- start start 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f399406cab0 0x7f3994153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39940cdac0 0x7f39940c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f399406cec0 con 0x7f399406cab0 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f39940cec10 con 0x7f399415e240 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.560+0000 7f3999ba8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39940b1580 con 0x7f39940cdac0 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.187:60610/0 (socket says 172.21.15.187:60610) 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 -- 172.21.15.187:0/1613382141 learned_addr learned my addr 172.21.15.187:0/1613382141 (peer_addr_for_me v2:172.21.15.187:0/0) 2024-04-25T14:01:46.562 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3992ffd640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39940cdac0 0x7f39940c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f39937fe640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f399406cab0 0x7f3994153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 -- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39940cdac0 msgr2=0x7f39940c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39940cdac0 0x7f39940c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 -- 172.21.15.187:0/1613382141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f399406cab0 msgr2=0x7f3994153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f399406cab0 0x7f3994153d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 -- 172.21.15.187:0/1613382141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3994154480 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3993fff640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f398c02ab30 tx=0x7f398c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c03f070 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398c031b40 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39940c9bb0 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398c044460 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39940c6830 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f398c044600 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.561+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f398c03ba70 con 0x7f399415e240 2024-04-25T14:01:46.563 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.562+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3994076a40 con 0x7f399415e240 2024-04-25T14:01:46.564 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.564+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f398c03bd50 con 0x7f399415e240 2024-04-25T14:01:46.637 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.638+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd new", "uuid": "98942485-6160-4b36-8909-62f908571405", "id": 7} v 0) -- 0x7f39940b2690 con 0x7f399415e240 2024-04-25T14:01:46.643 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.644+0000 7f397bfff640 1 -- 172.21.15.187:0/1613382141 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "98942485-6160-4b36-8909-62f908571405", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f398c044a30 con 0x7f399415e240 2024-04-25T14:01:46.644 INFO:teuthology.orchestra.run.smithi187.stdout:7 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 msgr2=0x7f39940c62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f398c02ab30 tx=0x7f398c004a40 comp rx=0 tx=0).stop 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 shutdown_connections 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f399415e240 0x7f39940c62f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f39940cdac0 0x7f39940c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 --2- 172.21.15.187:0/1613382141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f399406cab0 0x7f3994153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 >> 172.21.15.187:0/1613382141 conn(0x7f399409b4a0 msgr2=0x7f39940b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 shutdown_connections 2024-04-25T14:01:46.645 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:01:46.645+0000 7f3999ba8640 1 -- 172.21.15.187:0/1613382141 wait complete. 2024-04-25T14:01:46.656 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-25T14:01:46.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-25T14:01:46.658 INFO:tasks.ceph.osd.0:Started 2024-04-25T14:01:46.658 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-25T14:01:46.658 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-25T14:01:46.660 INFO:tasks.ceph.osd.1:Started 2024-04-25T14:01:46.660 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-25T14:01:46.660 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-25T14:01:46.662 INFO:tasks.ceph.osd.2:Started 2024-04-25T14:01:46.662 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-25T14:01:46.663 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-25T14:01:46.665 INFO:tasks.ceph.osd.3:Started 2024-04-25T14:01:46.665 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-25T14:01:46.665 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-25T14:01:46.667 INFO:tasks.ceph.osd.4:Started 2024-04-25T14:01:46.667 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-25T14:01:46.667 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-25T14:01:46.669 INFO:tasks.ceph.osd.5:Started 2024-04-25T14:01:46.669 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-25T14:01:46.669 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-25T14:01:46.671 INFO:tasks.ceph.osd.6:Started 2024-04-25T14:01:46.671 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-25T14:01:46.671 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-25T14:01:46.675 INFO:tasks.ceph.osd.7:Started 2024-04-25T14:01:46.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:01:46.681 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.683+0000 7fc041636040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T14:01:46.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.811+0000 7f4cd720c640 1 Processor -- start 2024-04-25T14:01:46.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.811+0000 7f4cd720c640 1 -- start start 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd720c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd01625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd720c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cd015f100 con 0x7f4cd00b5b10 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd720c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4cd015f3e0 con 0x7f4cd0158760 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd720c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd0070fd0 con 0x7f4cd015ec30 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccffff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4cd00b5b10 0x7f4cd015e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:33878/0 (socket says 172.21.15.150:33878) 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccffff640 1 -- 172.21.15.150:0/448952505 learned_addr learned my addr 172.21.15.150:0/448952505 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd4fa8640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f4cd0158760 0x7f4cd00b5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:58302/0 (socket says 172.21.15.150:58302) 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd57a9640 1 --2- 172.21.15.150:0/448952505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd01625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2050755114 0 0) 0x7f4cd015f3e0 con 0x7f4cd0158760 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cc0003520 con 0x7f4cd0158760 2024-04-25T14:01:46.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2802940111 0 0) 0x7f4cd015f100 con 0x7f4cd00b5b10 2024-04-25T14:01:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4cd015f3e0 con 0x7f4cd00b5b10 2024-04-25T14:01:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd57a9640 1 -- 172.21.15.150:0/448952505 >> v1:172.21.15.187:6789/0 conn(0x7f4cd0158760 legacy=0x7f4cd00b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd57a9640 1 -- 172.21.15.150:0/448952505 >> v1:172.21.15.150:6789/0 conn(0x7f4cd00b5b10 legacy=0x7f4cd015e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.813+0000 7f4cd57a9640 1 -- 172.21.15.150:0/448952505 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd0161b10 con 0x7f4cd015ec30 2024-04-25T14:01:46.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.814+0000 7f4cd57a9640 1 --2- 172.21.15.150:0/448952505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd01625f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009dc0 tx=0x7f4cc802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95efcad90c201311 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.814+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8007740 con 0x7f4cd015ec30 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.814+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc80078e0 con 0x7f4cd015ec30 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.814+0000 7f4ccf7fe640 1 -- 172.21.15.150:0/448952505 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8007cb0 con 0x7f4cd015ec30 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 -- 172.21.15.150:0/448952505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 msgr2=0x7f4cd01625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 --2- 172.21.15.150:0/448952505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd01625f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4cc8009dc0 tx=0x7f4cc802a6d0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 -- 172.21.15.150:0/448952505 shutdown_connections 2024-04-25T14:01:46.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 --2- 172.21.15.150:0/448952505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd01625f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 -- 172.21.15.150:0/448952505 >> 172.21.15.150:0/448952505 conn(0x7f4cd00d0400 msgr2=0x7f4cd00d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 -- 172.21.15.150:0/448952505 shutdown_connections 2024-04-25T14:01:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.815+0000 7f4cd720c640 1 -- 172.21.15.150:0/448952505 wait complete. 2024-04-25T14:01:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.817+0000 7f4cd720c640 1 Processor -- start 2024-04-25T14:01:46.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.817+0000 7f4cd720c640 1 -- start start 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 0x7f4cd015b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd0148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd00703d0 con 0x7f4cd01595d0 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4cd01620d0 con 0x7f4cd0158760 2024-04-25T14:01:46.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd720c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4cd00ce370 con 0x7f4cd015ec30 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd57a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd4fa8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 0x7f4cd015b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4ccffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd0148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd57a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51240/0 (socket says 172.21.15.150:51240) 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.818+0000 7f4cd57a9640 1 -- 172.21.15.150:0/1915875400 learned_addr learned my addr 172.21.15.150:0/1915875400 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 -- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 msgr2=0x7f4cd0148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd0148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 -- 172.21.15.150:0/1915875400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 msgr2=0x7f4cd01599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 -- 172.21.15.150:0/1915875400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cd015a2c0 con 0x7f4cd0158760 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd57a9640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd4fa8640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 0x7f4cd015b360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00cfe60 tx=0x7f4cc4001c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc400ee00 con 0x7f4cd0158760 2024-04-25T14:01:46.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4cc400e280 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4cc40072f0 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cd015a5a0 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.819+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cd015ab60 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.820+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4cc40047e0 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.820+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f4cc4019b40 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.820+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cd00cf5f0 con 0x7f4cd0158760 2024-04-25T14:01:46.821 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:46.818+0000 7fde84436040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T14:01:46.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.822+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4cc4038e50 con 0x7f4cd0158760 2024-04-25T14:01:46.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.884+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4cd00cfe90 con 0x7f4cd0158760 2024-04-25T14:01:46.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.884+0000 7f4ccd7fa640 1 -- 172.21.15.150:0/1915875400 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f4cd00cfe90 con 0x7f4cd0158760 2024-04-25T14:01:46.884 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:01:46.884 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":10,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:46.640223+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 msgr2=0x7f4cd015b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 0x7f4cd015b360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00cfe60 tx=0x7f4cc4001c60 comp rx=0 tx=0).stop 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 shutdown_connections 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4cd01595d0 0x7f4cd01599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4cd015ec30 0x7f4cd0148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 --2- 172.21.15.150:0/1915875400 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4cd0158760 0x7f4cd015b360 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:46.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.885+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 >> 172.21.15.150:0/1915875400 conn(0x7f4cd00d0400 msgr2=0x7f4cd00b8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:46.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.886+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 shutdown_connections 2024-04-25T14:01:46.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:01:46.886+0000 7f4cd720c640 1 -- 172.21.15.150:0/1915875400 wait complete. 2024-04-25T14:01:46.896 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-25T14:01:46.896 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-25T14:01:46.988 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:46.989+0000 7fc041636040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T14:01:47.042 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.043+0000 7fc041636040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T14:01:47.078 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.079+0000 7fde84436040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T14:01:47.131 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.131+0000 7fde84436040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T14:01:47.159 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.160+0000 7fc041636040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T14:01:47.246 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.246+0000 7fde84436040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T14:01:47.281 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-04-25T14:01:47.282+0000 7f25b43d6640 -1 Falling back to public interface 2024-04-25T14:01:47.292 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-25T14:01:47.292+0000 7fd3b6e16640 -1 Falling back to public interface 2024-04-25T14:01:47.293 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-04-25T14:01:47.294+0000 7f7c518dd640 -1 Falling back to public interface 2024-04-25T14:01:47.294 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-25T14:01:47.294+0000 7f29ecc25640 -1 Falling back to public interface 2024-04-25T14:01:47.300 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-25T14:01:47.300+0000 7f96244b3640 -1 Falling back to public interface 2024-04-25T14:01:47.305 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-04-25T14:01:47.306+0000 7fb01761c640 -1 Falling back to public interface 2024-04-25T14:01:47.306 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-25T14:01:47.306+0000 7f9328015640 -1 Falling back to public interface 2024-04-25T14:01:47.306 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-04-25T14:01:47.306+0000 7fb12bf44640 -1 Falling back to public interface 2024-04-25T14:01:47.446 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.447+0000 7fc041636040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T14:01:47.494 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.495+0000 7fc041636040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T14:01:47.529 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.530+0000 7fde84436040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T14:01:47.578 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.578+0000 7fde84436040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T14:01:47.652 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.653+0000 7fc041636040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T14:01:47.701 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T14:01:47.734 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.734+0000 7fde84436040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T14:01:47.795 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.796+0000 7fc041636040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T14:01:47.802 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.803+0000 7ff397dbe640 1 Processor -- start 2024-04-25T14:01:47.802 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.803+0000 7ff397dbe640 1 -- start start 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.804+0000 7ff397dbe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff39006df10 0x7ff39006e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.804+0000 7ff397dbe640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff39006e860 con 0x7ff39015c510 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.804+0000 7ff397dbe640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3900cf7f0 con 0x7ff3900d1f40 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.804+0000 7ff397dbe640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff3900d14a0 con 0x7ff39006df10 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff395359640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff39006df10 0x7ff39006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff39635b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff39015c510 0x7ff39015c920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47420/0 (socket says 172.21.15.150:47420) 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff39635b640 1 -- 172.21.15.150:0/283071567 learned_addr learned my addr 172.21.15.150:0/283071567 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff395359640 1 -- 172.21.15.150:0/283071567 >> v1:172.21.15.187:6789/0 conn(0x7ff3900d1f40 legacy=0x7ff3900b2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.804 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff395359640 1 -- 172.21.15.150:0/283071567 >> v1:172.21.15.150:6789/0 conn(0x7ff39015c510 legacy=0x7ff39015c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.805 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff395359640 1 -- 172.21.15.150:0/283071567 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3900b3f90 con 0x7ff39006df10 2024-04-25T14:01:47.805 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.805+0000 7ff395359640 1 --2- 172.21.15.150:0/283071567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39006e320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff38000ba40 tx=0x7ff38002c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ec91b5acf3b3dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:47.805 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.806+0000 7ff394b58640 1 -- 172.21.15.150:0/283071567 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38002f070 con 0x7ff39006df10 2024-04-25T14:01:47.805 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.806+0000 7ff394b58640 1 -- 172.21.15.150:0/283071567 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff380038070 con 0x7ff39006df10 2024-04-25T14:01:47.805 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.806+0000 7ff394b58640 1 -- 172.21.15.150:0/283071567 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380032a00 con 0x7ff39006df10 2024-04-25T14:01:47.806 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 -- 172.21.15.150:0/283071567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 msgr2=0x7ff39006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.806 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 --2- 172.21.15.150:0/283071567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39006e320 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff38000ba40 tx=0x7ff38002c8e0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.806 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 -- 172.21.15.150:0/283071567 shutdown_connections 2024-04-25T14:01:47.806 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 --2- 172.21.15.150:0/283071567 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39006e320 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.807 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 -- 172.21.15.150:0/283071567 >> 172.21.15.150:0/283071567 conn(0x7ff39009b4a0 msgr2=0x7ff39009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:47.807 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 -- 172.21.15.150:0/283071567 shutdown_connections 2024-04-25T14:01:47.807 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.807+0000 7ff397dbe640 1 -- 172.21.15.150:0/283071567 wait complete. 2024-04-25T14:01:47.809 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.810+0000 7ff397dbe640 1 Processor -- start 2024-04-25T14:01:47.809 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.810+0000 7ff397dbe640 1 -- start start 2024-04-25T14:01:47.812 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:47.812 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3900d1f40 0x7ff39009ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff39015c510 0x7ff3901e2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff3900b4830 con 0x7ff39006df10 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff3900b45d0 con 0x7ff3900d1f40 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff397dbe640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff3900b4b10 con 0x7ff39015c510 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff395b5a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff395b5a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46578/0 (socket says 172.21.15.150:46578) 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff395b5a640 1 -- 172.21.15.150:0/3885359110 learned_addr learned my addr 172.21.15.150:0/3885359110 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff39635b640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff39015c510 0x7ff3901e2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.812+0000 7ff395359640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3900d1f40 0x7ff39009ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 -- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff39015c510 msgr2=0x7ff3901e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff39015c510 0x7ff3901e2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 -- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3900d1f40 msgr2=0x7ff39009ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.813 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3900d1f40 0x7ff39009ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 -- 172.21.15.150:0/3885359110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff39009f930 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff395b5a640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c3f0 tx=0x7ff38c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c0052d0 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff38c005490 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c007260 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff39009fc10 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3900a01d0 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff38c007400 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.813+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7ff38c011c60 con 0x7ff39006df10 2024-04-25T14:01:47.814 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.814+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff390076a40 con 0x7ff39006df10 2024-04-25T14:01:47.815 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.816+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7ff38c0035f0 con 0x7ff39006df10 2024-04-25T14:01:47.850 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.851+0000 7fc041636040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T14:01:47.871 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.872+0000 7fde84436040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T14:01:47.877 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.878+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff3900b1680 con 0x7ff39006df10 2024-04-25T14:01:47.878 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.879+0000 7ff3867fc640 1 -- 172.21.15.150:0/3885359110 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7ff38c003a60 con 0x7ff39006df10 2024-04-25T14:01:47.878 INFO:teuthology.misc.health.smithi150.stdout: 2024-04-25T14:01:47.878 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":10,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:46.640223+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 msgr2=0x7ff39014ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff38c00c3f0 tx=0x7ff38c00c8b0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 shutdown_connections 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff39015c510 0x7ff3901e2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff3900d1f40 0x7ff39009ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 --2- 172.21.15.150:0/3885359110 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff39006df10 0x7ff39014ee20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:47.880 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 >> 172.21.15.150:0/3885359110 conn(0x7ff39009b4a0 msgr2=0x7ff39006ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:47.880 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 shutdown_connections 2024-04-25T14:01:47.880 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:47.880+0000 7ff397dbe640 1 -- 172.21.15.150:0/3885359110 wait complete. 2024-04-25T14:01:47.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T14:01:47.921 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.921+0000 7fde84436040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T14:01:47.924 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.926+0000 7fc041636040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T14:01:47.971 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:47.972+0000 7fc041636040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T14:01:47.990 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:47.991+0000 7fde84436040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T14:01:48.036 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:48.036+0000 7fde84436040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T14:01:48.457 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:48.458+0000 7fc041636040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T14:01:48.516 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:48.516+0000 7fde84436040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T14:01:48.575 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:48.576+0000 7fc041636040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T14:01:48.635 INFO:tasks.ceph.mgr.x.smithi187.stderr:2024-04-25T14:01:48.636+0000 7fde84436040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T14:01:49.589 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:49.590+0000 7fbfe1158640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-25T14:01:49.589 INFO:tasks.ceph.mgr.y.smithi150.stderr:2024-04-25T14:01:49.591+0000 7fbfe1158640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-25T14:01:50.760 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-04-25T14:01:50.761+0000 7f25b43d6640 -1 osd.3 0 log_to_monitors true 2024-04-25T14:01:50.810 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-04-25T14:01:50.811+0000 7fb12bf44640 -1 osd.0 0 log_to_monitors true 2024-04-25T14:01:50.811 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-04-25T14:01:50.812+0000 7f7c518dd640 -1 osd.2 0 log_to_monitors true 2024-04-25T14:01:50.818 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-04-25T14:01:50.819+0000 7fb01761c640 -1 osd.1 0 log_to_monitors true 2024-04-25T14:01:50.839 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-25T14:01:50.840+0000 7fd3b6e16640 -1 osd.5 0 log_to_monitors true 2024-04-25T14:01:50.845 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-25T14:01:50.846+0000 7f29ecc25640 -1 osd.6 0 log_to_monitors true 2024-04-25T14:01:50.852 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-25T14:01:50.853+0000 7f9328015640 -1 osd.4 0 log_to_monitors true 2024-04-25T14:01:50.857 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-25T14:01:50.857+0000 7f96244b3640 -1 osd.7 0 log_to_monitors true 2024-04-25T14:01:54.696 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T14:01:54.703 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-25T14:01:54.703+0000 7f9620449640 -1 osd.7 0 waiting for initial osdmap 2024-04-25T14:01:54.719 INFO:tasks.ceph.osd.7.smithi187.stderr:2024-04-25T14:01:54.720+0000 7f9618244640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.745 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-25T14:01:54.745+0000 7f29e8bbb640 -1 osd.6 0 waiting for initial osdmap 2024-04-25T14:01:54.755 INFO:tasks.ceph.osd.6.smithi187.stderr:2024-04-25T14:01:54.756+0000 7f29e09b6640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.807 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-25T14:01:54.808+0000 7f932392c640 -1 osd.4 0 waiting for initial osdmap 2024-04-25T14:01:54.819 INFO:tasks.ceph.osd.4.smithi187.stderr:2024-04-25T14:01:54.820+0000 7f931b727640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.821 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-04-25T14:01:54.822+0000 7fb012f0c640 -1 osd.1 0 waiting for initial osdmap 2024-04-25T14:01:54.838 INFO:tasks.ceph.osd.1.smithi150.stderr:2024-04-25T14:01:54.839+0000 7fb00ad07640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.851 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-04-25T14:01:54.852+0000 7f7c4d873640 -1 osd.2 0 waiting for initial osdmap 2024-04-25T14:01:54.854 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-04-25T14:01:54.855+0000 7f25b036c640 -1 osd.3 0 waiting for initial osdmap 2024-04-25T14:01:54.856 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.857+0000 7f80e085a640 1 Processor -- start 2024-04-25T14:01:54.857 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.858+0000 7f80e085a640 1 -- start start 2024-04-25T14:01:54.857 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-04-25T14:01:54.858+0000 7fb127eda640 -1 osd.0 0 waiting for initial osdmap 2024-04-25T14:01:54.858 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80e085a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.859 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80e085a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f80dc06e040 0x7f80dc06e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.859 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80e085a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80dc0d3130 con 0x7f80dc06e990 2024-04-25T14:01:54.859 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80e085a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f80dc0d0220 con 0x7f80dc06e040 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80e085a640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f80dc06cfa0 con 0x7f80dc0d27e0 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:56526/0 (socket says 172.21.15.150:56526) 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 -- 172.21.15.150:0/1776185417 learned_addr learned my addr 172.21.15.150:0/1776185417 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80d3fff640 1 --2- 172.21.15.150:0/1776185417 >> v2:172.21.15.150:3300/0 conn(0x7f80dc06e040 0x7f80dc06e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 -- 172.21.15.150:0/1776185417 >> v2:172.21.15.150:3300/0 conn(0x7f80dc06e040 msgr2=0x7f80dc06e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 --2- 172.21.15.150:0/1776185417 >> v2:172.21.15.150:3300/0 conn(0x7f80dc06e040 0x7f80dc06e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 -- 172.21.15.150:0/1776185417 >> v1:172.21.15.150:6789/0 conn(0x7f80dc06e990 legacy=0x7f80dc06eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:54.860 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.860+0000 7f80da59c640 1 -- 172.21.15.150:0/1776185417 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80dc06d280 con 0x7f80dc0d27e0 2024-04-25T14:01:54.861 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.861+0000 7f80da59c640 1 --2- 172.21.15.150:0/1776185417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80dc073850 tx=0x7f80cc02a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=266f183d75852500 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:54.866 INFO:tasks.ceph.osd.2.smithi150.stderr:2024-04-25T14:01:54.867+0000 7f7c4566e640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.866 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.867+0000 7f80d9d9b640 1 -- 172.21.15.150:0/1776185417 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc037070 con 0x7f80dc0d27e0 2024-04-25T14:01:54.866 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.867+0000 7f80d9d9b640 1 -- 172.21.15.150:0/1776185417 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80cc02ae50 con 0x7f80dc0d27e0 2024-04-25T14:01:54.866 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.867+0000 7f80d9d9b640 1 -- 172.21.15.150:0/1776185417 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80cc032860 con 0x7f80dc0d27e0 2024-04-25T14:01:54.867 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 -- 172.21.15.150:0/1776185417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 msgr2=0x7f80dc0d2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 --2- 172.21.15.150:0/1776185417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f80dc073850 tx=0x7f80cc02a300 comp rx=0 tx=0).stop 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 -- 172.21.15.150:0/1776185417 shutdown_connections 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 --2- 172.21.15.150:0/1776185417 >> v2:172.21.15.150:3300/0 conn(0x7f80dc06e040 0x7f80dc06e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 --2- 172.21.15.150:0/1776185417 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc0d2bf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 -- 172.21.15.150:0/1776185417 >> 172.21.15.150:0/1776185417 conn(0x7f80dc09b4a0 msgr2=0x7f80dc09b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 -- 172.21.15.150:0/1776185417 shutdown_connections 2024-04-25T14:01:54.868 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.868+0000 7f80e085a640 1 -- 172.21.15.150:0/1776185417 wait complete. 2024-04-25T14:01:54.869 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.870+0000 7f80e085a640 1 Processor -- start 2024-04-25T14:01:54.869 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.870+0000 7f80e085a640 1 -- start start 2024-04-25T14:01:54.872 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80dc06e040 0x7f80dc15dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.872 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.872 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc14bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc06db20 con 0x7f80dc06e990 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f80dc06d840 con 0x7f80dc0d27e0 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80e085a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f80dc0b3190 con 0x7f80dc06e040 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:46674/0 (socket says 172.21.15.150:46674) 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 -- 172.21.15.150:0/4249324486 learned_addr learned my addr 172.21.15.150:0/4249324486 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80da59c640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80dc06e040 0x7f80dc15dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80dad9d640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc14bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 -- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80dc06e040 msgr2=0x7f80dc15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80dc06e040 0x7f80dc15dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 -- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 msgr2=0x7f80dc14bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc14bd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:54.873 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 -- 172.21.15.150:0/4249324486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80dc0b2020 con 0x7f80dc06e990 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.872+0000 7f80d3fff640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e840 tx=0x7f80c400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.873+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c4017070 con 0x7f80dc06e990 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.873+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80c400c940 con 0x7f80dc06e990 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.873+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80dc149d80 con 0x7f80dc06e990 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.873+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c400cd10 con 0x7f80dc06e990 2024-04-25T14:01:54.874 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.873+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80dc15f8a0 con 0x7f80dc06e990 2024-04-25T14:01:54.875 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.874+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 4) ==== 96726+0+0 (secure 0 0 0) 0x7f80c401a070 con 0x7f80dc06e990 2024-04-25T14:01:54.875 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.874+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80dc15f690 con 0x7f80dc06e990 2024-04-25T14:01:54.875 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.875+0000 7f80d2ffd640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 0x7f80bc076930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:01:54.875 INFO:tasks.ceph.osd.0.smithi150.stderr:2024-04-25T14:01:54.875+0000 7fb11fcd5640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.875 INFO:tasks.ceph.osd.3.smithi150.stderr:2024-04-25T14:01:54.875+0000 7f25a8167640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:54.876 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.875+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f80c4094160 con 0x7f80dc06e990 2024-04-25T14:01:54.876 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.875+0000 7f80da59c640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 0x7f80bc076930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:01:54.876 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.875+0000 7f80da59c640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 0x7f80bc076930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f80cc034000 tx=0x7f80cc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:01:54.879 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:54.880+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f80dc0b2020 con 0x7f80dc06e990 2024-04-25T14:01:54.948 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-25T14:01:54.948+0000 7fd3b26a5640 -1 osd.5 0 waiting for initial osdmap 2024-04-25T14:01:54.964 INFO:tasks.ceph.osd.5.smithi187.stderr:2024-04-25T14:01:54.965+0000 7fd3aa4a0640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T14:01:55.059 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.059+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80dc076a40 con 0x7f80dc06e990 2024-04-25T14:01:55.059 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.060+0000 7f80d2ffd640 1 -- 172.21.15.150:0/4249324486 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f80c405f180 con 0x7f80dc06e990 2024-04-25T14:01:55.059 INFO:teuthology.misc.health.smithi150.stdout: 2024-04-25T14:01:55.060 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":12,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:52.590110+0000","last_up_change":"0.000000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:01:55.062 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 msgr2=0x7f80bc076930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:55.062 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 0x7f80bc076930 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f80cc034000 tx=0x7f80cc02e000 comp rx=0 tx=0).stop 2024-04-25T14:01:55.062 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 msgr2=0x7f80dc0b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:01:55.062 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f80c400e840 tx=0x7f80c400ed00 comp rx=0 tx=0).stop 2024-04-25T14:01:55.062 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 shutdown_connections 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f80bc074420 0x7f80bc076930 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f80dc0d27e0 0x7f80dc14bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f80dc06e990 0x7f80dc0b1ae0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 --2- 172.21.15.150:0/4249324486 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f80dc06e040 0x7f80dc15dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 >> 172.21.15.150:0/4249324486 conn(0x7f80dc09b4a0 msgr2=0x7f80dc0d0490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 shutdown_connections 2024-04-25T14:01:55.063 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:01:55.062+0000 7f80e085a640 1 -- 172.21.15.150:0/4249324486 wait complete. 2024-04-25T14:01:55.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-25T14:02:01.878 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:01.990 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.991+0000 7f1e6f015640 1 Processor -- start 2024-04-25T14:02:01.991 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.991+0000 7f1e6f015640 1 -- start start 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6f015640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e68163cf0 0x7f1e68161160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6f015640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e6815a530 0x7f1e680d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6f015640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e681616a0 con 0x7f1e6815a940 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6f015640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6806ee00 con 0x7f1e6815a530 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6f015640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e681642d0 con 0x7f1e68163cf0 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e67fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1e6815a530 0x7f1e680d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6d5b2640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1e6815a940 0x7f1e6806f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44894/0 (socket says 172.21.15.150:44894) 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6d5b2640 1 -- 172.21.15.150:0/1835674449 learned_addr learned my addr 172.21.15.150:0/1835674449 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e6cdb1640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e68163cf0 0x7f1e68161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:01.993 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e677fe640 1 -- 172.21.15.150:0/1835674449 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2037637535 0 0) 0x7f1e681616a0 con 0x7f1e6815a940 2024-04-25T14:02:01.994 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e677fe640 1 -- 172.21.15.150:0/1835674449 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e58003520 con 0x7f1e6815a940 2024-04-25T14:02:01.994 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e67fff640 1 -- 172.21.15.150:0/1835674449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e68163cf0 msgr2=0x7f1e68161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:01.994 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e67fff640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e68163cf0 0x7f1e68161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:01.994 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e67fff640 1 -- 172.21.15.150:0/1835674449 >> v1:172.21.15.150:6789/0 conn(0x7f1e6815a940 legacy=0x7f1e6806f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:01.994 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.993+0000 7f1e67fff640 1 -- 172.21.15.150:0/1835674449 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e681645b0 con 0x7f1e6815a530 2024-04-25T14:02:01.996 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.994+0000 7f1e67fff640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e6815a530 0x7f1e680d02f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e540097a0 tx=0x7f1e5402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daebbb75e9150328 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:01.996 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.994+0000 7f1e677fe640 1 -- 172.21.15.150:0/1835674449 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54037070 con 0x7f1e6815a530 2024-04-25T14:02:01.996 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.994+0000 7f1e677fe640 1 -- 172.21.15.150:0/1835674449 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e54032440 con 0x7f1e6815a530 2024-04-25T14:02:01.996 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.994+0000 7f1e677fe640 1 -- 172.21.15.150:0/1835674449 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54032810 con 0x7f1e6815a530 2024-04-25T14:02:01.996 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 -- 172.21.15.150:0/1835674449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e6815a530 msgr2=0x7f1e680d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e6815a530 0x7f1e680d02f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1e540097a0 tx=0x7f1e5402a180 comp rx=0 tx=0).stop 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 -- 172.21.15.150:0/1835674449 shutdown_connections 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e6815a530 0x7f1e680d02f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 --2- 172.21.15.150:0/1835674449 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e68163cf0 0x7f1e68161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 -- 172.21.15.150:0/1835674449 >> 172.21.15.150:0/1835674449 conn(0x7f1e68071530 msgr2=0x7f1e68071940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 -- 172.21.15.150:0/1835674449 shutdown_connections 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.995+0000 7f1e6f015640 1 -- 172.21.15.150:0/1835674449 wait complete. 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.997+0000 7f1e6f015640 1 Processor -- start 2024-04-25T14:02:01.997 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.997+0000 7f1e6f015640 1 -- start start 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e6815a530 0x7f1e681f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e6815a940 0x7f1e680c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1e6806f9a0 con 0x7f1e68163cf0 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1e68164b70 con 0x7f1e6815a530 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6f015640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1e6806fb20 con 0x7f1e6815a940 2024-04-25T14:02:02.001 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.998+0000 7f1e6d5b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:55980/0 (socket says 172.21.15.150:55980) 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 -- 172.21.15.150:0/461750056 learned_addr learned my addr 172.21.15.150:0/461750056 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e67fff640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e6815a940 0x7f1e680c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6cdb1640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e6815a530 0x7f1e681f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 -- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e6815a940 msgr2=0x7f1e680c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e6815a940 0x7f1e680c7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 -- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e6815a530 msgr2=0x7f1e681f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e6815a530 0x7f1e681f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 -- 172.21.15.150:0/461750056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e680c82c0 con 0x7f1e68163cf0 2024-04-25T14:02:02.002 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6d5b2640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000caa0 tx=0x7f1e6000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60017070 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6000ac00 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e680c8aa0 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e60012400 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:01.999+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e680c9060 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.000+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e68163a70 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.001+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1e6001a070 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.001+0000 7f1e657fa640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 0x7f1e4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.001+0000 7f1e6cdb1640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 0x7f1e4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.002+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f1e60094bb0 con 0x7f1e68163cf0 2024-04-25T14:02:02.003 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.002+0000 7f1e6cdb1640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 0x7f1e4c0769f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e68074660 tx=0x7f1e5c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.005 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.006+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1e68163a70 con 0x7f1e68163cf0 2024-04-25T14:02:02.181 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.182+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e68076a40 con 0x7f1e68163cf0 2024-04-25T14:02:02.182 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.183+0000 7f1e657fa640 1 -- 172.21.15.150:0/461750056 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f1e6005f040 con 0x7f1e68163cf0 2024-04-25T14:02:02.182 INFO:teuthology.misc.health.smithi150.stdout: 2024-04-25T14:02:02.182 INFO:teuthology.misc.health.smithi150.stdout:{"epoch":16,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:58.715236+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:02.184 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 msgr2=0x7f1e4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.184 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 0x7f1e4c0769f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e68074660 tx=0x7f1e5c008040 comp rx=0 tx=0).stop 2024-04-25T14:02:02.184 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 msgr2=0x7f1e680c7a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e6000caa0 tx=0x7f1e6000cf60 comp rx=0 tx=0).stop 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 shutdown_connections 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1e4c0744e0 0x7f1e4c0769f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1e68163cf0 0x7f1e680c7a60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1e6815a940 0x7f1e680c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 --2- 172.21.15.150:0/461750056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1e6815a530 0x7f1e681f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 >> 172.21.15.150:0/461750056 conn(0x7f1e68071530 msgr2=0x7f1e680d3810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 shutdown_connections 2024-04-25T14:02:02.185 INFO:teuthology.misc.health.smithi150.stderr:2024-04-25T14:02:02.185+0000 7f1e6f015640 1 -- 172.21.15.150:0/461750056 wait complete. 2024-04-25T14:02:02.194 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-25T14:02:02.195 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-25T14:02:02.195 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T14:02:02.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T14:02:02.197 INFO:tasks.ceph.mds.a:Started 2024-04-25T14:02:02.197 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T14:02:02.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T14:02:02.199 INFO:tasks.ceph.mds.c:Started 2024-04-25T14:02:02.199 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T14:02:02.199 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T14:02:02.202 INFO:tasks.ceph.mds.b:Started 2024-04-25T14:02:02.202 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T14:02:02.202 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T14:02:02.204 INFO:tasks.ceph.mds.d:Started 2024-04-25T14:02:02.205 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-25T14:02:02.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:02.281 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:02:02.282 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-04-25T14:02:02.298 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:02:02.298 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:02:02.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.320+0000 7f59c1ab1640 1 Processor -- start 2024-04-25T14:02:02.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.320+0000 7f59c1ab1640 1 -- start start 2024-04-25T14:02:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59c1ab1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0b4f60 0x7f59bc0cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59c1ab1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59bc161160 0x7f59bc15cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59bc161160 0x7f59bc15cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59baffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f59bc0cfe30 0x7f59bc160a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44922/0 (socket says 172.21.15.150:44922) 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59bc161160 0x7f59bc15cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56016/0 (socket says 172.21.15.150:56016) 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59baffd640 1 -- 172.21.15.150:0/1971967464 learned_addr learned my addr 172.21.15.150:0/1971967464 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59c1ab1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59bc0b3f00 con 0x7f59bc0cfe30 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f59bc0b4ad0 con 0x7f59bc161160 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f59bc0b41a0 con 0x7f59bc0b4f60 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bb7fe640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0b4f60 0x7f59bc0cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59ba7fc640 1 -- 172.21.15.150:0/1971967464 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2126553945 0 0) 0x7f59bc0b3f00 con 0x7f59bc0cfe30 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59ba7fc640 1 -- 172.21.15.150:0/1971967464 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59a4003520 con 0x7f59bc0cfe30 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 -- 172.21.15.150:0/1971967464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0b4f60 msgr2=0x7f59bc0cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0b4f60 0x7f59bc0cf8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 -- 172.21.15.150:0/1971967464 >> v1:172.21.15.150:6789/0 conn(0x7f59bc0cfe30 legacy=0x7f59bc160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.322+0000 7f59bbfff640 1 -- 172.21.15.150:0/1971967464 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59bc0b4480 con 0x7f59bc161160 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.323+0000 7f59bbfff640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc161160 0x7f59bc15cfe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009d90 tx=0x7f59b402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c20e08d9470ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.323+0000 7f59ba7fc640 1 -- 172.21.15.150:0/1971967464 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4032790 con 0x7f59bc161160 2024-04-25T14:02:02.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.323+0000 7f59ba7fc640 1 -- 172.21.15.150:0/1971967464 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b4038070 con 0x7f59bc161160 2024-04-25T14:02:02.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.323+0000 7f59ba7fc640 1 -- 172.21.15.150:0/1971967464 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b4032b50 con 0x7f59bc161160 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc161160 msgr2=0x7f59bc15cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc161160 0x7f59bc15cfe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009d90 tx=0x7f59b402c890 comp rx=0 tx=0).stop 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 shutdown_connections 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc161160 0x7f59bc15cfe0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/1971967464 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0b4f60 0x7f59bc0cf8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 >> 172.21.15.150:0/1971967464 conn(0x7f59bc09b4a0 msgr2=0x7f59bc09b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 shutdown_connections 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.324+0000 7f59c1ab1640 1 -- 172.21.15.150:0/1971967464 wait complete. 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.326+0000 7f59c1ab1640 1 Processor -- start 2024-04-25T14:02:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.326+0000 7f59c1ab1640 1 -- start start 2024-04-25T14:02:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59bc0b4f60 0x7f59bc0d2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0d2700 0x7f59bc158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56020/0 (socket says 172.21.15.150:56020) 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 -- 172.21.15.150:0/891996155 learned_addr learned my addr 172.21.15.150:0/891996155 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc1618b0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f59bc161730 con 0x7f59bc0b4f60 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f59bc15d520 con 0x7f59bc0d2700 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59bb7fe640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59bc0b4f60 0x7f59bc0d2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59bbfff640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0d2700 0x7f59bc158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 -- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0d2700 msgr2=0x7f59bc158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0d2700 0x7f59bc158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 -- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59bc0b4f60 msgr2=0x7f59bc0d2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59bc0b4f60 0x7f59bc0d2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.327+0000 7f59baffd640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59bc0d2b10 con 0x7f59bc0cfe30 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59baffd640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c390 tx=0x7f59b0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b0017070 con 0x7f59bc0cfe30 2024-04-25T14:02:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59b0005ce0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59bc0d0a50 con 0x7f59bc0cfe30 2024-04-25T14:02:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59b00072d0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.328+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59bc0afac0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.329+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59bc076a40 con 0x7f59bc0cfe30 2024-04-25T14:02:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.330+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f59b001a070 con 0x7f59bc0cfe30 2024-04-25T14:02:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.330+0000 7f59a3fff640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 0x7f59a8076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.330+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f59b00940f0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.330+0000 7f59bb7fe640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 0x7f59a8076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.330+0000 7f59bb7fe640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 0x7f59a8076a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59bc0734e0 tx=0x7f59ac009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.335+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f59bc0d2b10 con 0x7f59bc0cfe30 2024-04-25T14:02:02.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.510+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59bc0d07e0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.511+0000 7f59a3fff640 1 -- 172.21.15.150:0/891996155 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f59b005e5f0 con 0x7f59bc0cfe30 2024-04-25T14:02:02.510 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:02.510 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:58.715236+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:02.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 msgr2=0x7f59a8076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 0x7f59a8076a10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f59bc0734e0 tx=0x7f59ac009110 comp rx=0 tx=0).stop 2024-04-25T14:02:02.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 msgr2=0x7f59bc0d21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59b000c390 tx=0x7f59b0001e70 comp rx=0 tx=0).stop 2024-04-25T14:02:02.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 shutdown_connections 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59a8074500 0x7f59a8076a10 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f59bc0d2700 0x7f59bc158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59bc0cfe30 0x7f59bc0d21c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 --2- 172.21.15.150:0/891996155 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59bc0b4f60 0x7f59bc0d2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 >> 172.21.15.150:0/891996155 conn(0x7f59bc09b4a0 msgr2=0x7f59bc0cd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 shutdown_connections 2024-04-25T14:02:02.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.513+0000 7f59c1ab1640 1 -- 172.21.15.150:0/891996155 wait complete. 2024-04-25T14:02:02.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:02:02.524 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:02.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.634+0000 7f6ec6451640 1 Processor -- start 2024-04-25T14:02:02.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.634+0000 7f6ec6451640 1 -- start start 2024-04-25T14:02:02.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec6451640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ec00d1c20 0x7f6ec00b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec6451640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 0x7f6ec006ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec6451640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ec0163eb0 con 0x7f6ec01602f0 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec6451640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6ec0164ac0 con 0x7f6ec00d1c20 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec6451640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6ec0164190 con 0x7f6ec00b5970 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ec00d1c20 0x7f6ec00b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ec00d1c20 0x7f6ec00b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56024/0 (socket says 172.21.15.150:56024) 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 -- 172.21.15.150:0/582465440 learned_addr learned my addr 172.21.15.150:0/582465440 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec49ee640 1 --2- 172.21.15.150:0/582465440 >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 0x7f6ec006ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 -- 172.21.15.150:0/582465440 >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 msgr2=0x7f6ec006ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 --2- 172.21.15.150:0/582465440 >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 0x7f6ec006ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 -- 172.21.15.150:0/582465440 >> v1:172.21.15.187:6789/0 conn(0x7f6ec01602f0 legacy=0x7f6ec00d1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ebf7fe640 1 -- 172.21.15.150:0/582465440 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec0164470 con 0x7f6ec00d1c20 2024-04-25T14:02:02.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.636+0000 7f6ec49ee640 1 --2- 172.21.15.150:0/582465440 >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 0x7f6ec006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.637+0000 7f6ebf7fe640 1 --2- 172.21.15.150:0/582465440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec00d1c20 0x7f6ec00b5430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00c73c0 tx=0x7f6eb402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe08c317ea94087 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.637+0000 7f6ebeffd640 1 -- 172.21.15.150:0/582465440 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4037070 con 0x7f6ec00d1c20 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.637+0000 7f6ebeffd640 1 -- 172.21.15.150:0/582465440 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb4032440 con 0x7f6ec00d1c20 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.637+0000 7f6ebeffd640 1 -- 172.21.15.150:0/582465440 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4032810 con 0x7f6ec00d1c20 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 -- 172.21.15.150:0/582465440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec00d1c20 msgr2=0x7f6ec00b5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 --2- 172.21.15.150:0/582465440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec00d1c20 0x7f6ec00b5430 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6ec00c73c0 tx=0x7f6eb402a270 comp rx=0 tx=0).stop 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 -- 172.21.15.150:0/582465440 shutdown_connections 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 --2- 172.21.15.150:0/582465440 >> v2:172.21.15.187:3300/0 conn(0x7f6ec00b5970 0x7f6ec006ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 --2- 172.21.15.150:0/582465440 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec00d1c20 0x7f6ec00b5430 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 -- 172.21.15.150:0/582465440 >> 172.21.15.150:0/582465440 conn(0x7f6ec00d03c0 msgr2=0x7f6ec00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 -- 172.21.15.150:0/582465440 shutdown_connections 2024-04-25T14:02:02.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.638+0000 7f6ec6451640 1 -- 172.21.15.150:0/582465440 wait complete. 2024-04-25T14:02:02.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.640+0000 7f6ec6451640 1 Processor -- start 2024-04-25T14:02:02.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.640+0000 7f6ec6451640 1 -- start start 2024-04-25T14:02:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 0x7f6ec015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ec01602f0 0x7f6ec014cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00ce600 con 0x7f6ec015ece0 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec00b5d80 con 0x7f6ec01602f0 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec6451640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6ec00ce780 con 0x7f6ec00b5970 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec49ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 0x7f6ec015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec49ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56036/0 (socket says 172.21.15.150:56036) 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ec49ee640 1 -- 172.21.15.150:0/1890209279 learned_addr learned my addr 172.21.15.150:0/1890209279 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebf7fe640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ec01602f0 0x7f6ec014cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebffff640 1 -- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ec01602f0 msgr2=0x7f6ec014cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebffff640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ec01602f0 0x7f6ec014cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebffff640 1 -- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 msgr2=0x7f6ec015cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.641+0000 7f6ebffff640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ebffff640 1 -- 172.21.15.150:0/1890209279 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec00b82d0 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ec49ee640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ebffff640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 0x7f6ec015bf10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000c3c0 tx=0x7f6eb000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00052d0 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eb0005490 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb0007260 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec00b85b0 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.642+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec00b8b70 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.643+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ec0076a40 con 0x7f6ec00b5970 2024-04-25T14:02:02.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.643+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6eb0007400 con 0x7f6ec00b5970 2024-04-25T14:02:02.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.644+0000 7f6ebcff9640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 0x7f6eac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.644+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f6eb00941c0 con 0x7f6ec00b5970 2024-04-25T14:02:02.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.644+0000 7f6ebf7fe640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 0x7f6eac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.644+0000 7f6ebf7fe640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 0x7f6eac0769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4034000 tx=0x7f6eb402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.649+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6eb005e640 con 0x7f6ec00b5970 2024-04-25T14:02:02.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.822+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ec00b98c0 con 0x7f6ec00b5970 2024-04-25T14:02:02.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.823+0000 7f6ebcff9640 1 -- 172.21.15.150:0/1890209279 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f6eb0062940 con 0x7f6ec00b5970 2024-04-25T14:02:02.822 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:02.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 msgr2=0x7f6eac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 0x7f6eac0769c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4034000 tx=0x7f6eb402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 msgr2=0x7f6ec015bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 0x7f6ec015bf10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6eb000c3c0 tx=0x7f6eb000c880 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 shutdown_connections 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6eac0744b0 0x7f6eac0769c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ec015ece0 0x7f6ec015cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ec01602f0 0x7f6ec014cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 --2- 172.21.15.150:0/1890209279 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ec00b5970 0x7f6ec015bf10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 >> 172.21.15.150:0/1890209279 conn(0x7f6ec00d03c0 msgr2=0x7f6ec00d3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 shutdown_connections 2024-04-25T14:02:02.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.825+0000 7f6ec6451640 1 -- 172.21.15.150:0/1890209279 wait complete. 2024-04-25T14:02:02.835 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:02.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.942+0000 7fa08a17d640 1 Processor -- start 2024-04-25T14:02:02.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.942+0000 7fa08a17d640 1 -- start start 2024-04-25T14:02:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa08a17d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0840d13c0 0x7fa084160790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa08a17d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0840d4510 con 0x7fa084163c40 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa08a17d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0840d47f0 con 0x7fa0840d0fb0 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa08a17d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa0840d55b0 con 0x7fa0840d13c0 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa083fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0840d13c0 0x7fa084160790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa0837fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa084163c40 0x7fa0840d08a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44924/0 (socket says 172.21.15.150:44924) 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa0837fe640 1 -- 172.21.15.150:0/4076079317 learned_addr learned my addr 172.21.15.150:0/4076079317 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa083fff640 1 -- 172.21.15.150:0/4076079317 >> v1:172.21.15.187:6789/0 conn(0x7fa0840d0fb0 legacy=0x7fa084160080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa083fff640 1 -- 172.21.15.150:0/4076079317 >> v1:172.21.15.150:6789/0 conn(0x7fa084163c40 legacy=0x7fa0840d08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa0827fc640 1 -- 172.21.15.150:0/4076079317 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3480446204 0 0) 0x7fa0840d4510 con 0x7fa084163c40 2024-04-25T14:02:02.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.944+0000 7fa083fff640 1 -- 172.21.15.150:0/4076079317 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0840d4ad0 con 0x7fa0840d13c0 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.945+0000 7fa083fff640 1 --2- 172.21.15.150:0/4076079317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d13c0 0x7fa084160790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32aef2b82cc70879 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.945+0000 7fa0827fc640 1 -- 172.21.15.150:0/4076079317 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c031600 con 0x7fa0840d13c0 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.945+0000 7fa0827fc640 1 -- 172.21.15.150:0/4076079317 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c0317a0 con 0x7fa0840d13c0 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.945+0000 7fa0827fc640 1 -- 172.21.15.150:0/4076079317 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c031b70 con 0x7fa0840d13c0 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 -- 172.21.15.150:0/4076079317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d13c0 msgr2=0x7fa084160790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 --2- 172.21.15.150:0/4076079317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d13c0 0x7fa084160790 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009d90 tx=0x7fa07c02a610 comp rx=0 tx=0).stop 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 -- 172.21.15.150:0/4076079317 shutdown_connections 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 --2- 172.21.15.150:0/4076079317 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d13c0 0x7fa084160790 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 -- 172.21.15.150:0/4076079317 >> 172.21.15.150:0/4076079317 conn(0x7fa0840b7650 msgr2=0x7fa0840b7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 -- 172.21.15.150:0/4076079317 shutdown_connections 2024-04-25T14:02:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.946+0000 7fa08a17d640 1 -- 172.21.15.150:0/4076079317 wait complete. 2024-04-25T14:02:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.947+0000 7fa08a17d640 1 Processor -- start 2024-04-25T14:02:02.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.948+0000 7fa08a17d640 1 -- start start 2024-04-25T14:02:02.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d0fb0 0x7fa0841f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840d13c0 0x7fa0841f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa08406d580 con 0x7fa084163c40 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840d5090 con 0x7fa0840d0fb0 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa08a17d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0840d41a0 con 0x7fa0840d13c0 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56052/0 (socket says 172.21.15.150:56052) 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 -- 172.21.15.150:0/2893691298 learned_addr learned my addr 172.21.15.150:0/2893691298 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa0837fe640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d0fb0 0x7fa0841f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa082ffd640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840d13c0 0x7fa0841f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 -- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840d13c0 msgr2=0x7fa0841f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840d13c0 0x7fa0841f6750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 -- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d0fb0 msgr2=0x7fa0841f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d0fb0 0x7fa0841f5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.949+0000 7fa083fff640 1 -- 172.21.15.150:0/2893691298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0841f6320 con 0x7fa084163c40 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa083fff640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009eb0 tx=0x7fa07c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c040070 con 0x7fa084163c40 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c030b60 con 0x7fa084163c40 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c045460 con 0x7fa084163c40 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0840cddd0 con 0x7fa084163c40 2024-04-25T14:02:02.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.950+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0840ce390 con 0x7fa084163c40 2024-04-25T14:02:02.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.951+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0840ce770 con 0x7fa084163c40 2024-04-25T14:02:02.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.951+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa07c039040 con 0x7fa084163c40 2024-04-25T14:02:02.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.952+0000 7fa06bfff640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 0x7fa0700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:02.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.952+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa07c0b66b0 con 0x7fa084163c40 2024-04-25T14:02:02.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.952+0000 7fa0837fe640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 0x7fa0700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:02.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.952+0000 7fa0837fe640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 0x7fa0700769c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa074000f60 tx=0x7fa074007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:02.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:02.956+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa0840ce770 con 0x7fa084163c40 2024-04-25T14:02:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.169+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa084076a40 con 0x7fa084163c40 2024-04-25T14:02:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.170+0000 7fa06bfff640 1 -- 172.21.15.150:0/2893691298 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1395 (secure 0 0 0) 0x7fa07c081250 con 0x7fa084163c40 2024-04-25T14:02:03.169 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:03.169 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T14:02:03.169 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 2 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.171+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 msgr2=0x7fa0700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 0x7fa0700769c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa074000f60 tx=0x7fa074007c00 comp rx=0 tx=0).stop 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 msgr2=0x7fa0841f6c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa07c009eb0 tx=0x7fa07c004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 shutdown_connections 2024-04-25T14:02:03.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0700744b0 0x7fa0700769c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa084163c40 0x7fa0841f6c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840d13c0 0x7fa0841f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 --2- 172.21.15.150:0/2893691298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d0fb0 0x7fa0841f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 >> 172.21.15.150:0/2893691298 conn(0x7fa0840b7650 msgr2=0x7fa0840b7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 shutdown_connections 2024-04-25T14:02:03.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:03.172+0000 7fa08a17d640 1 -- 172.21.15.150:0/2893691298 wait complete. 2024-04-25T14:02:05.183 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:05.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.304+0000 7f9130fee640 1 Processor -- start 2024-04-25T14:02:05.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.304+0000 7f9130fee640 1 -- start start 2024-04-25T14:02:05.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9130fee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f912c0b5ab0 0x7f912c0cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9130fee640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f912c0b8700 0x7f912c0b8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9130fee640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f912c0b7f40 con 0x7f912c163690 2024-04-25T14:02:05.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9130fee640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f912c164bc0 con 0x7f912c0b5ab0 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9130fee640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f912c0d0180 con 0x7f912c0b8700 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f912c0b5ab0 0x7f912c0cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f912c0b5ab0 0x7f912c0cfc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56066/0 (socket says 172.21.15.150:56066) 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 -- 172.21.15.150:0/3871459638 learned_addr learned my addr 172.21.15.150:0/3871459638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f912ad9d640 1 --2- 172.21.15.150:0/3871459638 >> v2:172.21.15.187:3300/0 conn(0x7f912c0b8700 0x7f912c0b8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 -- 172.21.15.150:0/3871459638 >> v2:172.21.15.187:3300/0 conn(0x7f912c0b8700 msgr2=0x7f912c0b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 --2- 172.21.15.150:0/3871459638 >> v2:172.21.15.187:3300/0 conn(0x7f912c0b8700 0x7f912c0b8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 -- 172.21.15.150:0/3871459638 >> v1:172.21.15.187:6789/0 conn(0x7f912c163690 legacy=0x7f912c0b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.306+0000 7f9129d9b640 1 -- 172.21.15.150:0/3871459638 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c164620 con 0x7f912c0b5ab0 2024-04-25T14:02:05.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.307+0000 7f9129d9b640 1 --2- 172.21.15.150:0/3871459638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c0b5ab0 0x7f912c0cfc40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f912c08cfa0 tx=0x7f912002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6beeaa08575c4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.307+0000 7f912959a640 1 -- 172.21.15.150:0/3871459638 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120037070 con 0x7f912c0b5ab0 2024-04-25T14:02:05.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.307+0000 7f912959a640 1 -- 172.21.15.150:0/3871459638 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9120032440 con 0x7f912c0b5ab0 2024-04-25T14:02:05.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.307+0000 7f912959a640 1 -- 172.21.15.150:0/3871459638 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9120032810 con 0x7f912c0b5ab0 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 -- 172.21.15.150:0/3871459638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c0b5ab0 msgr2=0x7f912c0cfc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 --2- 172.21.15.150:0/3871459638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c0b5ab0 0x7f912c0cfc40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f912c08cfa0 tx=0x7f912002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 -- 172.21.15.150:0/3871459638 shutdown_connections 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 --2- 172.21.15.150:0/3871459638 >> v2:172.21.15.187:3300/0 conn(0x7f912c0b8700 0x7f912c0b8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 --2- 172.21.15.150:0/3871459638 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c0b5ab0 0x7f912c0cfc40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 -- 172.21.15.150:0/3871459638 >> 172.21.15.150:0/3871459638 conn(0x7f912c0d03a0 msgr2=0x7f912c0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 -- 172.21.15.150:0/3871459638 shutdown_connections 2024-04-25T14:02:05.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.308+0000 7f9130fee640 1 -- 172.21.15.150:0/3871459638 wait complete. 2024-04-25T14:02:05.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.310+0000 7f9130fee640 1 Processor -- start 2024-04-25T14:02:05.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.310+0000 7f9130fee640 1 -- start start 2024-04-25T14:02:05.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 0x7f912c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f912c0b8700 0x7f912c0b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f912c0b81e0 con 0x7f912c163690 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f912c1648e0 con 0x7f912c0b8700 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9130fee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f912c0b2ca0 con 0x7f912c0b5ab0 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f912ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f912ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56076/0 (socket says 172.21.15.150:56076) 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f912ad9d640 1 -- 172.21.15.150:0/1722232225 learned_addr learned my addr 172.21.15.150:0/1722232225 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.311+0000 7f9129d9b640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f912c0b8700 0x7f912c0b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912a59c640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 0x7f912c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 -- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 msgr2=0x7f912c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 0x7f912c0706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 -- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f912c0b8700 msgr2=0x7f912c0b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f912c0b8700 0x7f912c0b3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 -- 172.21.15.150:0/1722232225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f912c0b4970 con 0x7f912c163690 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912a59c640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 0x7f912c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f912ad9d640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9124007d80 tx=0x7f912400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124014420 con 0x7f912c163690 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9124019070 con 0x7f912c163690 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f912c144af0 con 0x7f912c163690 2024-04-25T14:02:05.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9124014800 con 0x7f912c163690 2024-04-25T14:02:05.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.312+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f912c1450b0 con 0x7f912c163690 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.313+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f912c0b46e0 con 0x7f912c163690 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.314+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91240149a0 con 0x7f912c163690 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.314+0000 7f911affd640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 0x7f910c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.314+0000 7f912a59c640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 0x7f910c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.314+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f9124095620 con 0x7f912c163690 2024-04-25T14:02:05.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.315+0000 7f912a59c640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 0x7f910c0769f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f911c000f60 tx=0x7f911c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.319+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9124060190 con 0x7f912c163690 2024-04-25T14:02:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.532+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f912c076a40 con 0x7f912c163690 2024-04-25T14:02:05.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.533+0000 7f911affd640 1 -- 172.21.15.150:0/1722232225 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3873 (secure 0 0 0) 0x7f912405fa90 con 0x7f912c163690 2024-04-25T14:02:05.536 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:05.536 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4293,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-25T14:02:05.537 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 3 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 msgr2=0x7f910c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 0x7f910c0769f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f911c000f60 tx=0x7f911c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 msgr2=0x7f912c146a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9124007d80 tx=0x7f912400e4b0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 shutdown_connections 2024-04-25T14:02:05.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f910c0744e0 0x7f910c0769f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f912c163690 0x7f912c146a70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f912c0b8700 0x7f912c0b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 --2- 172.21.15.150:0/1722232225 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f912c0b5ab0 0x7f912c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 >> 172.21.15.150:0/1722232225 conn(0x7f912c0d03a0 msgr2=0x7f912c160ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:05.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 shutdown_connections 2024-04-25T14:02:05.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.535+0000 7f9130fee640 1 -- 172.21.15.150:0/1722232225 wait complete. 2024-04-25T14:02:05.545 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.657+0000 7f3c91630640 1 Processor -- start 2024-04-25T14:02:05.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.657+0000 7f3c91630640 1 -- start start 2024-04-25T14:02:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c91630640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c8c1601d0 0x7f3c8c0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c91630640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3c8c06da00 0x7f3c8c163ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c91630640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c8c06c570 con 0x7f3c8c0b9070 2024-04-25T14:02:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c91630640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3c8c1605e0 con 0x7f3c8c1601d0 2024-04-25T14:02:05.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c91630640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3c8c0d5980 con 0x7f3c8c06da00 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c8c1601d0 0x7f3c8c0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c8c1601d0 0x7f3c8c0d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56080/0 (socket says 172.21.15.150:56080) 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 -- 172.21.15.150:0/2392500596 learned_addr learned my addr 172.21.15.150:0/2392500596 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8b7fe640 1 --2- 172.21.15.150:0/2392500596 >> v2:172.21.15.187:3300/0 conn(0x7f3c8c06da00 0x7f3c8c163ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 -- 172.21.15.150:0/2392500596 >> v2:172.21.15.187:3300/0 conn(0x7f3c8c06da00 msgr2=0x7f3c8c163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 --2- 172.21.15.150:0/2392500596 >> v2:172.21.15.187:3300/0 conn(0x7f3c8c06da00 0x7f3c8c163ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 -- 172.21.15.150:0/2392500596 >> v1:172.21.15.187:6789/0 conn(0x7f3c8c0b9070 legacy=0x7f3c8c06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.659+0000 7f3c8affd640 1 -- 172.21.15.150:0/2392500596 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c06c810 con 0x7f3c8c1601d0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.660+0000 7f3c8affd640 1 --2- 172.21.15.150:0/2392500596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c1601d0 0x7f3c8c0d5100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0d3ce0 tx=0x7f3c7c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af227d4446150423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.660+0000 7f3c89ffb640 1 -- 172.21.15.150:0/2392500596 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c036070 con 0x7f3c8c1601d0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.660+0000 7f3c89ffb640 1 -- 172.21.15.150:0/2392500596 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c7c031600 con 0x7f3c8c1601d0 2024-04-25T14:02:05.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.660+0000 7f3c89ffb640 1 -- 172.21.15.150:0/2392500596 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c0319d0 con 0x7f3c8c1601d0 2024-04-25T14:02:05.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 -- 172.21.15.150:0/2392500596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c1601d0 msgr2=0x7f3c8c0d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 --2- 172.21.15.150:0/2392500596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c1601d0 0x7f3c8c0d5100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c0d3ce0 tx=0x7f3c7c02a640 comp rx=0 tx=0).stop 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 -- 172.21.15.150:0/2392500596 shutdown_connections 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 --2- 172.21.15.150:0/2392500596 >> v2:172.21.15.187:3300/0 conn(0x7f3c8c06da00 0x7f3c8c163ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 --2- 172.21.15.150:0/2392500596 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c1601d0 0x7f3c8c0d5100 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 -- 172.21.15.150:0/2392500596 >> 172.21.15.150:0/2392500596 conn(0x7f3c8c0d5d90 msgr2=0x7f3c8c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 -- 172.21.15.150:0/2392500596 shutdown_connections 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.661+0000 7f3c91630640 1 -- 172.21.15.150:0/2392500596 wait complete. 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.663+0000 7f3c91630640 1 Processor -- start 2024-04-25T14:02:05.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.663+0000 7f3c91630640 1 -- start start 2024-04-25T14:02:05.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c8c06da00 0x7f3c8c0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c8c1601d0 0x7f3c8c158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c0d5640 con 0x7f3c8c0b9070 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3c8c06cdd0 con 0x7f3c8c06da00 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c91630640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3c8c06de10 con 0x7f3c8c1601d0 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56094/0 (socket says 172.21.15.150:56094) 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 -- 172.21.15.150:0/913911183 learned_addr learned my addr 172.21.15.150:0/913911183 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8affd640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c8c06da00 0x7f3c8c0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 -- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c8c1601d0 msgr2=0x7f3c8c158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c8c1601d0 0x7f3c8c158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 -- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c8c06da00 msgr2=0x7f3c8c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c8c06da00 0x7f3c8c0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 -- 172.21.15.150:0/913911183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c8c1f7a50 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.664+0000 7f3c8a7fc640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b3e0 tx=0x7f3c8000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.665+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80015020 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.665+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c80011830 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.665+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c8c0ced20 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.665+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c80011c20 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.665+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c8c0cd250 con 0x7f3c8c0b9070 2024-04-25T14:02:05.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.666+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c8c0cea90 con 0x7f3c8c0b9070 2024-04-25T14:02:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.666+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3c80018020 con 0x7f3c8c0b9070 2024-04-25T14:02:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.667+0000 7f3c737fe640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 0x7f3c780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.667+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3c80094da0 con 0x7f3c8c0b9070 2024-04-25T14:02:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.667+0000 7f3c8affd640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 0x7f3c780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.667+0000 7f3c8affd640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 0x7f3c780769c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c033000 tx=0x7f3c7c009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.671+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c8001f020 con 0x7f3c8c0b9070 2024-04-25T14:02:05.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.852+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c8c076a40 con 0x7f3c8c0b9070 2024-04-25T14:02:05.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.853+0000 7f3c737fe640 1 -- 172.21.15.150:0/913911183 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12884 (secure 0 0 0) 0x7f3c8005f890 con 0x7f3c8c0b9070 2024-04-25T14:02:05.852 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:05.852 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:01:58.715236+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:05.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 msgr2=0x7f3c780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 0x7f3c780769c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c033000 tx=0x7f3c7c009000 comp rx=0 tx=0).stop 2024-04-25T14:02:05.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 msgr2=0x7f3c8c1f7510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3c8000b3e0 tx=0x7f3c8000b8a0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 shutdown_connections 2024-04-25T14:02:05.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c780744b0 0x7f3c780769c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c8c1601d0 0x7f3c8c158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c8c0b9070 0x7f3c8c1f7510 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 --2- 172.21.15.150:0/913911183 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c8c06da00 0x7f3c8c0cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 >> 172.21.15.150:0/913911183 conn(0x7f3c8c0d5d90 msgr2=0x7f3c8c0b5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:05.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 shutdown_connections 2024-04-25T14:02:05.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.855+0000 7f3c91630640 1 -- 172.21.15.150:0/913911183 wait complete. 2024-04-25T14:02:05.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:02:05.866 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:05.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.978+0000 7f3e24ee7640 1 Processor -- start 2024-04-25T14:02:05.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.978+0000 7f3e24ee7640 1 -- start start 2024-04-25T14:02:05.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e24ee7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e200d3de0 0x7f3e201600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e24ee7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e200d1750 con 0x7f3e20160630 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e24ee7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e200d55d0 con 0x7f3e200d1000 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e24ee7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3e20160a40 con 0x7f3e200d3de0 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1e59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e200d3de0 0x7f3e201600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1dd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3e20160630 0x7f3e200d08f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44926/0 (socket says 172.21.15.150:44926) 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1dd9b640 1 -- 172.21.15.150:0/1858731573 learned_addr learned my addr 172.21.15.150:0/1858731573 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1e59c640 1 -- 172.21.15.150:0/1858731573 >> v1:172.21.15.187:6789/0 conn(0x7f3e200d1000 legacy=0x7f3e2006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1e59c640 1 -- 172.21.15.150:0/1858731573 >> v1:172.21.15.150:6789/0 conn(0x7f3e20160630 legacy=0x7f3e200d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1e59c640 1 -- 172.21.15.150:0/1858731573 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e200d4530 con 0x7f3e200d3de0 2024-04-25T14:02:05.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.980+0000 7f3e1d59a640 1 -- 172.21.15.150:0/1858731573 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3449310367 0 0) 0x7f3e200d1750 con 0x7f3e20160630 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.981+0000 7f3e1e59c640 1 --2- 172.21.15.150:0/1858731573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200d3de0 0x7f3e201600f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009d90 tx=0x7f3e1002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96adf7549418a71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.981+0000 7f3e1d59a640 1 -- 172.21.15.150:0/1858731573 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10007740 con 0x7f3e200d3de0 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.981+0000 7f3e1d59a640 1 -- 172.21.15.150:0/1858731573 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e100078e0 con 0x7f3e200d3de0 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.981+0000 7f3e1d59a640 1 -- 172.21.15.150:0/1858731573 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e10007cb0 con 0x7f3e200d3de0 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1858731573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200d3de0 msgr2=0x7f3e201600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1858731573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200d3de0 0x7f3e201600f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3e10009d90 tx=0x7f3e1002a6a0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1858731573 shutdown_connections 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1858731573 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200d3de0 0x7f3e201600f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1858731573 >> 172.21.15.150:0/1858731573 conn(0x7f3e200d25c0 msgr2=0x7f3e200d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1858731573 shutdown_connections 2024-04-25T14:02:05.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.982+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1858731573 wait complete. 2024-04-25T14:02:05.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.984+0000 7f3e24ee7640 1 Processor -- start 2024-04-25T14:02:05.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.984+0000 7f3e24ee7640 1 -- start start 2024-04-25T14:02:05.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 0x7f3e2006d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e20160630 0x7f3e200ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e200d4dd0 con 0x7f3e200cedc0 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3e200d4af0 con 0x7f3e20160630 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e24ee7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e200d50b0 con 0x7f3e200d3de0 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56124/0 (socket says 172.21.15.150:56124) 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 -- 172.21.15.150:0/1721226771 learned_addr learned my addr 172.21.15.150:0/1721226771 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1dd9b640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e20160630 0x7f3e200ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1e59c640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 0x7f3e2006d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 -- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 msgr2=0x7f3e2006d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 0x7f3e2006d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 -- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e20160630 msgr2=0x7f3e200ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e20160630 0x7f3e200ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1ed9d640 1 -- 172.21.15.150:0/1721226771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e200cf1d0 con 0x7f3e200cedc0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.985+0000 7f3e1e59c640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 0x7f3e2006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e1ed9d640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b410 tx=0x7f3e1800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18015030 con 0x7f3e200cedc0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e18004020 con 0x7f3e200cedc0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e200ccc60 con 0x7f3e200cedc0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e18004410 con 0x7f3e200cedc0 2024-04-25T14:02:05.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.986+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e200cd220 con 0x7f3e200cedc0 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.987+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e20076a40 con 0x7f3e200cedc0 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.988+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3e18018070 con 0x7f3e200cedc0 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.988+0000 7f3e0effd640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 0x7f3e000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.988+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f3e180948d0 con 0x7f3e200cedc0 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.988+0000 7f3e1e59c640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 0x7f3e000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:05.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.988+0000 7f3e1e59c640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 0x7f3e000769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e10034000 tx=0x7f3e1002f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:05.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:05.993+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e200cf1d0 con 0x7f3e200cedc0 2024-04-25T14:02:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.170+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3e2015e540 con 0x7f3e200cedc0 2024-04-25T14:02:06.170 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.170+0000 7f3e0effd640 1 -- 172.21.15.150:0/1721226771 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f3e1805f3b0 con 0x7f3e200cedc0 2024-04-25T14:02:06.170 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 msgr2=0x7f3e000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 0x7f3e000769c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3e10034000 tx=0x7f3e1002f000 comp rx=0 tx=0).stop 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 msgr2=0x7f3e201f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b410 tx=0x7f3e1800b8d0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 shutdown_connections 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e000744b0 0x7f3e000769c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e200cedc0 0x7f3e201f6350 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e20160630 0x7f3e200ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 --2- 172.21.15.150:0/1721226771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e200d3de0 0x7f3e2006d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 >> 172.21.15.150:0/1721226771 conn(0x7f3e200d25c0 msgr2=0x7f3e2006ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:06.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 shutdown_connections 2024-04-25T14:02:06.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.172+0000 7f3e24ee7640 1 -- 172.21.15.150:0/1721226771 wait complete. 2024-04-25T14:02:06.183 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T14:02:06.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.291+0000 7f7a6c292640 1 Processor -- start 2024-04-25T14:02:06.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.291+0000 7f7a6c292640 1 -- start start 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6c292640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a64160e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6c292640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a641613a0 con 0x7f7a640b4f60 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6c292640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a640b3f00 con 0x7f7a6415cfe0 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6c292640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a640b4ad0 con 0x7f7a64160a50 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6a02e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7a640b4f60 0x7f7a640cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44938/0 (socket says 172.21.15.150:44938) 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6a02e640 1 -- 172.21.15.150:0/1508299148 learned_addr learned my addr 172.21.15.150:0/1508299148 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707636888 0 0) 0x7f7a5c001610 con 0x7f7a640b4f60 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a4c003520 con 0x7f7a640b4f60 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6982d640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a64160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1942906263 0 0) 0x7f7a4c003520 con 0x7f7a640b4f60 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 msgr2=0x7f7a64160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a64160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 >> v1:172.21.15.187:6789/0 conn(0x7f7a6415cfe0 legacy=0x7f7a6415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a640b41a0 con 0x7f7a640b4f60 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.293+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f7a5c001d40 con 0x7f7a640b4f60 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 >> v1:172.21.15.150:6789/0 conn(0x7f7a640b4f60 legacy=0x7f7a640cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a4c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a4c004800 0x7f7a4c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a4c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7a640b4ad0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4c00baf0 con 0x7f7a4c004800 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4c00bcf0 con 0x7f7a64160a50 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a4c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6982d640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a4c004800 0x7f7a4c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a82f640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a4c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 -- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 msgr2=0x7f7a4c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a4c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 -- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a4c004800 msgr2=0x7f7a4c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:06.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a4c004800 0x7f7a4c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a02e640 1 -- 172.21.15.150:0/1508299148 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a4c003520 con 0x7f7a4c0039e0 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.294+0000 7f7a6a82f640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a64160a50 0x7f7a4c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a4c003e50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c002230 tx=0x7f7a4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b6a4bf3b1c1675 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c00a2f0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5c02f070 con 0x7f7a4c0039e0 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6902c640 1 -- 172.21.15.150:0/1508299148 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c02b660 con 0x7f7a4c0039e0 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6c292640 1 -- 172.21.15.150:0/1508299148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 msgr2=0x7f7a4c003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6c292640 1 --2- 172.21.15.150:0/1508299148 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a4c003e50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c002230 tx=0x7f7a4c005830 comp rx=0 tx=0).stop 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6a82f640 1 -- 172.21.15.150:0/1508299148 reap_dead start 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6c292640 1 -- 172.21.15.150:0/1508299148 shutdown_connections 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.295+0000 7f7a6c292640 1 -- 172.21.15.150:0/1508299148 >> 172.21.15.150:0/1508299148 conn(0x7f7a6409b4a0 msgr2=0x7f7a6409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.296+0000 7f7a6c292640 1 -- 172.21.15.150:0/1508299148 shutdown_connections 2024-04-25T14:02:06.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.296+0000 7f7a6c292640 1 -- 172.21.15.150:0/1508299148 wait complete. 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.297+0000 7f7a6c292640 1 Processor -- start 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.297+0000 7f7a6c292640 1 -- start start 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.298+0000 7f7a6c292640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.298+0000 7f7a6c292640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a641f63d0 0x7f7a641f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 0x7f7a641f8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7a640d0000 con 0x7f7a4c0039e0 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7a640b4760 con 0x7f7a641f6f20 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7a64161730 con 0x7f7a641f63d0 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56142/0 (socket says 172.21.15.150:56142) 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 -- 172.21.15.150:0/2533651597 learned_addr learned my addr 172.21.15.150:0/2533651597 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6982d640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a641f63d0 0x7f7a641f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a82f640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 0x7f7a641f8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 -- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a641f63d0 msgr2=0x7f7a641f75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a641f63d0 0x7f7a641f75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 -- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 msgr2=0x7f7a641f8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 0x7f7a641f8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 -- 172.21.15.150:0/2533651597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a641f7e20 con 0x7f7a4c0039e0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a82f640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 0x7f7a641f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6a02e640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c002230 tx=0x7f7a5c02d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c02ba70 con 0x7f7a4c0039e0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5c02bc10 con 0x7f7a4c0039e0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a641f8100 con 0x7f7a4c0039e0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c001610 con 0x7f7a4c0039e0 2024-04-25T14:02:06.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.299+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a641f86c0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.300+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a64076a40 con 0x7f7a4c0039e0 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.301+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7a5c047020 con 0x7f7a4c0039e0 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.301+0000 7f7a62ffd640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 0x7f7a500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.302+0000 7f7a6982d640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 0x7f7a500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.302+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f7a5c0b4fe0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.302+0000 7f7a6982d640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 0x7f7a500769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a640c5580 tx=0x7f7a58008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.306+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7a641613a0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.486+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f7a640cebd0 con 0x7f7a4c0039e0 2024-04-25T14:02:06.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.486+0000 7f7a62ffd640 1 -- 172.21.15.150:0/2533651597 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f7a5c07fb80 con 0x7f7a4c0039e0 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:02:06.486 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-04-25T14:02:06.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 msgr2=0x7f7a500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 0x7f7a500769f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7a640c5580 tx=0x7f7a58008040 comp rx=0 tx=0).stop 2024-04-25T14:02:06.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 msgr2=0x7f7a641f4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c002230 tx=0x7f7a5c02d070 comp rx=0 tx=0).stop 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 shutdown_connections 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7a500744e0 0x7f7a500769f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7a641f6f20 0x7f7a641f8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7a641f63d0 0x7f7a641f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 --2- 172.21.15.150:0/2533651597 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7a4c0039e0 0x7f7a641f4070 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 >> 172.21.15.150:0/2533651597 conn(0x7f7a6409b4a0 msgr2=0x7f7a640b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 shutdown_connections 2024-04-25T14:02:06.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.488+0000 7f7a6c292640 1 -- 172.21.15.150:0/2533651597 wait complete. 2024-04-25T14:02:06.498 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T14:02:06.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T14:02:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.611+0000 7fa13ce1a640 1 Processor -- start 2024-04-25T14:02:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.611+0000 7fa13ce1a640 1 -- start start 2024-04-25T14:02:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13ce1a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa13815e110 0x7fa1380b6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13ce1a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa1380734f0 0x7fa1380d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13ce1a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1380cec00 con 0x7fa1380ce220 2024-04-25T14:02:06.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13ce1a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa1380ceee0 con 0x7fa13815e110 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13ce1a640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa1380cf1c0 con 0x7fa1380734f0 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa13815e110 0x7fa1380b6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa13815e110 0x7fa1380b6ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56148/0 (socket says 172.21.15.150:56148) 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 -- 172.21.15.150:0/2787220992 learned_addr learned my addr 172.21.15.150:0/2787220992 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa136d9d640 1 --2- 172.21.15.150:0/2787220992 >> v2:172.21.15.187:3300/0 conn(0x7fa1380734f0 0x7fa1380d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 -- 172.21.15.150:0/2787220992 >> v2:172.21.15.187:3300/0 conn(0x7fa1380734f0 msgr2=0x7fa1380d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 --2- 172.21.15.150:0/2787220992 >> v2:172.21.15.187:3300/0 conn(0x7fa1380734f0 0x7fa1380d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 -- 172.21.15.150:0/2787220992 >> v1:172.21.15.187:6789/0 conn(0x7fa1380ce220 legacy=0x7fa138072de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.613+0000 7fa13659c640 1 -- 172.21.15.150:0/2787220992 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1380d2040 con 0x7fa13815e110 2024-04-25T14:02:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.614+0000 7fa13659c640 1 --2- 172.21.15.150:0/2787220992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa13815e110 0x7fa1380b6ca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa138074d60 tx=0x7fa12c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b792cf6c3097445b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.614+0000 7fa13559a640 1 -- 172.21.15.150:0/2787220992 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c037070 con 0x7fa13815e110 2024-04-25T14:02:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.614+0000 7fa13559a640 1 -- 172.21.15.150:0/2787220992 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c032440 con 0x7fa13815e110 2024-04-25T14:02:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.614+0000 7fa13559a640 1 -- 172.21.15.150:0/2787220992 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c032810 con 0x7fa13815e110 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 -- 172.21.15.150:0/2787220992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa13815e110 msgr2=0x7fa1380b6ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/2787220992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa13815e110 0x7fa1380b6ca0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa138074d60 tx=0x7fa12c02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 -- 172.21.15.150:0/2787220992 shutdown_connections 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/2787220992 >> v2:172.21.15.187:3300/0 conn(0x7fa1380734f0 0x7fa1380d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/2787220992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa13815e110 0x7fa1380b6ca0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 -- 172.21.15.150:0/2787220992 >> 172.21.15.150:0/2787220992 conn(0x7fa1380d39c0 msgr2=0x7fa1380d3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 -- 172.21.15.150:0/2787220992 shutdown_connections 2024-04-25T14:02:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.615+0000 7fa13ce1a640 1 -- 172.21.15.150:0/2787220992 wait complete. 2024-04-25T14:02:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.617+0000 7fa13ce1a640 1 Processor -- start 2024-04-25T14:02:06.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.617+0000 7fa13ce1a640 1 -- start start 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa13815e110 0x7fa13814ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 0x7fa13806f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa138161d50 con 0x7fa1380ce220 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa138161bd0 con 0x7fa13814f840 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13ce1a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1380ce630 con 0x7fa13815e110 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.618+0000 7fa13659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56150/0 (socket says 172.21.15.150:56150) 2024-04-25T14:02:06.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 -- 172.21.15.150:0/3053945447 learned_addr learned my addr 172.21.15.150:0/3053945447 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa136d9d640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 0x7fa13806f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa135d9b640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa13815e110 0x7fa13814ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 -- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa13815e110 msgr2=0x7fa13814ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa13815e110 0x7fa13814ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 -- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 msgr2=0x7fa13806f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 0x7fa13806f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 -- 172.21.15.150:0/3053945447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa13806e6f0 con 0x7fa1380ce220 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa136d9d640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 0x7fa13806f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13659c640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa12c02a1e0 tx=0x7fa12c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c041020 con 0x7fa1380ce220 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa12c0306c0 con 0x7fa1380ce220 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa12c037040 con 0x7fa1380ce220 2024-04-25T14:02:06.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa13806e9d0 con 0x7fa1380ce220 2024-04-25T14:02:06.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.619+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa13814fc80 con 0x7fa1380ce220 2024-04-25T14:02:06.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.620+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa138076a40 con 0x7fa1380ce220 2024-04-25T14:02:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.621+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa12c044020 con 0x7fa1380ce220 2024-04-25T14:02:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.622+0000 7fa126ffd640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 0x7fa1180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.622+0000 7fa135d9b640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 0x7fa1180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.622+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fa12c0b5730 con 0x7fa1380ce220 2024-04-25T14:02:06.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.622+0000 7fa135d9b640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 0x7fa1180769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:06.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.626+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa12c04b030 con 0x7fa1380ce220 2024-04-25T14:02:06.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:06.802+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa1380d1b50 con 0x7fa1380ce220 2024-04-25T14:02:07.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.118+0000 7fa126ffd640 1 -- 172.21.15.150:0/3053945447 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7fa12c0802a0 con 0x7fa1380ce220 2024-04-25T14:02:07.118 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-04-25T14:02:07.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 msgr2=0x7fa1180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 0x7fa1180769c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa128006fd0 tx=0x7fa128008040 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 msgr2=0x7fa1380c74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa12c02a1e0 tx=0x7fa12c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 shutdown_connections 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1180744b0 0x7fa1180769c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa13814f840 0x7fa13806f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa13815e110 0x7fa13814ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 --2- 172.21.15.150:0/3053945447 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1380ce220 0x7fa1380c74f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 >> 172.21.15.150:0/3053945447 conn(0x7fa1380d39c0 msgr2=0x7fa1380b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 shutdown_connections 2024-04-25T14:02:07.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.121+0000 7fa13ce1a640 1 -- 172.21.15.150:0/3053945447 wait complete. 2024-04-25T14:02:07.135 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-25T14:02:07.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.250+0000 7f1d3e59d640 1 Processor -- start 2024-04-25T14:02:07.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.250+0000 7f1d3e59d640 1 -- start start 2024-04-25T14:02:07.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3e59d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d38070830 0x7f1d380d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3e59d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d380d15d0 con 0x7f1d3815e360 2024-04-25T14:02:07.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3e59d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d380ce440 con 0x7f1d380cf3d0 2024-04-25T14:02:07.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3e59d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d380d1870 con 0x7f1d38070830 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3cb3a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1d3815e360 0x7f1d380d25b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:44944/0 (socket says 172.21.15.150:44944) 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d3cb3a640 1 -- 172.21.15.150:0/3536064253 learned_addr learned my addr 172.21.15.150:0/3536064253 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d38070830 0x7f1d380d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2577675572 0 0) 0x7f1d380d15d0 con 0x7f1d3815e360 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.251+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d20003520 con 0x7f1d3815e360 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3070516401 0 0) 0x7f1d380ce440 con 0x7f1d380cf3d0 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d380d15d0 con 0x7f1d380cf3d0 2024-04-25T14:02:07.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3421681426 0 0) 0x7f1d20003520 con 0x7f1d3815e360 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d38070830 msgr2=0x7f1d380d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d38070830 0x7f1d380d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 >> v1:172.21.15.187:6789/0 conn(0x7f1d380cf3d0 legacy=0x7f1d38070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d380d1b50 con 0x7f1d3815e360 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f1d300019f0 con 0x7f1d3815e360 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d38070830 0x7f1d380d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:07.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 >> v1:172.21.15.150:6789/0 conn(0x7f1d3815e360 legacy=0x7f1d380d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20003b70 0x7f1d20003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d20004800 0x7f1d20004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d200051a0 0x7f1d2000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d380d1870 con 0x7f1d20004800 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1d2000bf00 con 0x7f1d20003b70 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d2000c0b0 con 0x7f1d200051a0 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d20004800 0x7f1d20004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d3cb3a640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d200051a0 0x7f1d2000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d37fff640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20003b70 0x7f1d20003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 -- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d200051a0 msgr2=0x7f1d2000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d200051a0 0x7f1d2000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 -- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20003b70 msgr2=0x7f1d20003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20003b70 0x7f1d20003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d377fe640 1 -- 172.21.15.150:0/3536064253 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d20003520 con 0x7f1d20004800 2024-04-25T14:02:07.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.252+0000 7f1d37fff640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20003b70 0x7f1d20003fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.253+0000 7f1d377fe640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d20004800 0x7f1d20004c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009c90 tx=0x7f1d20005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5dcdc4bb695abc5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.253+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c036070 con 0x7f1d20004800 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.253+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c031400 con 0x7f1d20004800 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.253+0000 7f1d36ffd640 1 -- 172.21.15.150:0/3536064253 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c009860 con 0x7f1d20004800 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 -- 172.21.15.150:0/3536064253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d20004800 msgr2=0x7f1d20004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/3536064253 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d20004800 0x7f1d20004c10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c009c90 tx=0x7f1d20005830 comp rx=0 tx=0).stop 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3cb3a640 1 -- 172.21.15.150:0/3536064253 reap_dead start 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 -- 172.21.15.150:0/3536064253 shutdown_connections 2024-04-25T14:02:07.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 -- 172.21.15.150:0/3536064253 >> 172.21.15.150:0/3536064253 conn(0x7f1d380d3960 msgr2=0x7f1d380d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 -- 172.21.15.150:0/3536064253 shutdown_connections 2024-04-25T14:02:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.254+0000 7f1d3e59d640 1 -- 172.21.15.150:0/3536064253 wait complete. 2024-04-25T14:02:07.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.255+0000 7f1d3e59d640 1 Processor -- start 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.256+0000 7f1d3e59d640 1 -- start start 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20004800 0x7f1d381f1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d380b4c30 0x7f1d380cca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1d38070eb0 con 0x7f1d381f4890 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1d38070d30 con 0x7f1d20004800 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d3e59d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1d380cdfb0 con 0x7f1d380b4c30 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56164/0 (socket says 172.21.15.150:56164) 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 -- 172.21.15.150:0/1121622548 learned_addr learned my addr 172.21.15.150:0/1121622548 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 -- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d380b4c30 msgr2=0x7f1d380cca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d380b4c30 0x7f1d380cca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d37fff640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20004800 0x7f1d381f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 -- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20004800 msgr2=0x7f1d381f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20004800 0x7f1d381f1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 -- 172.21.15.150:0/1121622548 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d380b5040 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.257+0000 7f1d377fe640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000c00 tx=0x7f1d2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.258+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c040020 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.258+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d2c0093e0 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.258+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d381f77e0 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.258+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c036040 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.258+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d381f7da0 con 0x7f1d381f4890 2024-04-25T14:02:07.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.259+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d38076a40 con 0x7f1d381f4890 2024-04-25T14:02:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.259+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1d2c043020 con 0x7f1d381f4890 2024-04-25T14:02:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.260+0000 7f1d34ff9640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 0x7f1d240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.260+0000 7f1d37fff640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 0x7f1d240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.260+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f1d2c0b5680 con 0x7f1d381f4890 2024-04-25T14:02:07.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.260+0000 7f1d37fff640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 0x7f1d240769c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d28001630 tx=0x7f1d28009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:07.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.265+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d2c04a030 con 0x7f1d381f4890 2024-04-25T14:02:07.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:07.440+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f1d3815f7c0 con 0x7f1d381f4890 2024-04-25T14:02:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.120+0000 7f1d34ff9640 1 -- 172.21.15.150:0/1121622548 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) ==== 171+0+0 (secure 0 0 0) 0x7f1d2c0b50d0 con 0x7f1d381f4890 2024-04-25T14:02:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 msgr2=0x7f1d240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 0x7f1d240769c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1d28001630 tx=0x7f1d28009040 comp rx=0 tx=0).stop 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 msgr2=0x7f1d380b46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c000c00 tx=0x7f1d2c004730 comp rx=0 tx=0).stop 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 shutdown_connections 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d240744b0 0x7f1d240769c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1d380b4c30 0x7f1d380cca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1d381f4890 0x7f1d380b46f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 --2- 172.21.15.150:0/1121622548 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1d20004800 0x7f1d381f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.122+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 >> 172.21.15.150:0/1121622548 conn(0x7f1d380d3960 msgr2=0x7f1d380b3ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:08.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.123+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 shutdown_connections 2024-04-25T14:02:08.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.123+0000 7f1d3e59d640 1 -- 172.21.15.150:0/1121622548 wait complete. 2024-04-25T14:02:08.146 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T14:02:08.155 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:02:08.192 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:02:08.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.261+0000 7fe594134640 1 Processor -- start 2024-04-25T14:02:08.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.261+0000 7fe594134640 1 -- start start 2024-04-25T14:02:08.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe594134640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe58c09bbb0 0x7fe58c160340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe594134640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c0d2f10 0x7fe58c0b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe594134640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe58c06f640 con 0x7fe58c0b2350 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe594134640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe58c0b0170 con 0x7fe58c09bbb0 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe594134640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe58c0b0410 con 0x7fe58c0d2f10 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe5916cf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe58c09bbb0 0x7fe58c160340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe5916cf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe58c09bbb0 0x7fe58c160340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36472/0 (socket says 172.21.15.150:36472) 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.262+0000 7fe5916cf640 1 -- 172.21.15.150:0/652135619 learned_addr learned my addr 172.21.15.150:0/652135619 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5926d1640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c0d2f10 0x7fe58c0b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5916cf640 1 -- 172.21.15.150:0/652135619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c0d2f10 msgr2=0x7fe58c0b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:08.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5916cf640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c0d2f10 0x7fe58c0b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5916cf640 1 -- 172.21.15.150:0/652135619 >> v1:172.21.15.187:6789/0 conn(0x7fe58c0b2350 legacy=0x7fe58c09b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5916cf640 1 -- 172.21.15.150:0/652135619 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c0b06f0 con 0x7fe58c09bbb0 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe5916cf640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c09bbb0 0x7fe58c160340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009870 tx=0x7fe57c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=593406e7e8941929 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.263+0000 7fe590ece640 1 -- 172.21.15.150:0/652135619 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c037070 con 0x7fe58c09bbb0 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe590ece640 1 -- 172.21.15.150:0/652135619 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe57c032440 con 0x7fe58c09bbb0 2024-04-25T14:02:08.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe590ece640 1 -- 172.21.15.150:0/652135619 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c032810 con 0x7fe58c09bbb0 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 -- 172.21.15.150:0/652135619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c09bbb0 msgr2=0x7fe58c160340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c09bbb0 0x7fe58c160340 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009870 tx=0x7fe57c02a270 comp rx=0 tx=0).stop 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 -- 172.21.15.150:0/652135619 shutdown_connections 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c0d2f10 0x7fe58c0b2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 --2- 172.21.15.150:0/652135619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c09bbb0 0x7fe58c160340 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 -- 172.21.15.150:0/652135619 >> 172.21.15.150:0/652135619 conn(0x7fe58c0b76d0 msgr2=0x7fe58c0b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:08.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 -- 172.21.15.150:0/652135619 shutdown_connections 2024-04-25T14:02:08.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.264+0000 7fe594134640 1 -- 172.21.15.150:0/652135619 wait complete. 2024-04-25T14:02:08.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.266+0000 7fe594134640 1 Processor -- start 2024-04-25T14:02:08.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.266+0000 7fe594134640 1 -- start start 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 0x7fe58c1421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c1f9b30 0x7fe58c1f9f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c0d3600 con 0x7fe58c0b2350 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe58c0d3320 con 0x7fe58c0d2f10 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe594134640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe58c0b3330 con 0x7fe58c1f9b30 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.267+0000 7fe591ed0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36480/0 (socket says 172.21.15.150:36480) 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 -- 172.21.15.150:0/1474211554 learned_addr learned my addr 172.21.15.150:0/1474211554 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5926d1640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c1f9b30 0x7fe58c1f9f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5916cf640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 0x7fe58c1421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 -- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c1f9b30 msgr2=0x7fe58c1f9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c1f9b30 0x7fe58c1f9f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 -- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 msgr2=0x7fe58c1421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 0x7fe58c1421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 -- 172.21.15.150:0/1474211554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe58c1fa480 con 0x7fe58c0b2350 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5916cf640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 0x7fe58c1421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe591ed0640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c400 tx=0x7fe58800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe588017070 con 0x7fe58c0b2350 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5880052d0 con 0x7fe58c0b2350 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe58c06d750 con 0x7fe58c0b2350 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe588003830 con 0x7fe58c0b2350 2024-04-25T14:02:08.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.268+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe58c153620 con 0x7fe58c0b2350 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.269+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe58c076a40 con 0x7fe58c0b2350 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.270+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe58801a070 con 0x7fe58c0b2350 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.270+0000 7fe5827fc640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 0x7fe5700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.270+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7fe588094480 con 0x7fe58c0b2350 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.270+0000 7fe5916cf640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 0x7fe5700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:08.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.271+0000 7fe5916cf640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 0x7fe5700769f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe57c034000 tx=0x7fe57c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:08.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.275+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe58c1fa480 con 0x7fe58c0b2350 2024-04-25T14:02:08.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:08.488+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe58c071180 con 0x7fe58c0b2350 2024-04-25T14:02:09.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.150+0000 7fe5827fc640 1 -- 172.21.15.150:0/1474211554 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fe58805e630 con 0x7fe58c0b2350 2024-04-25T14:02:09.151 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T14:02:09.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.154+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 msgr2=0x7fe5700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.155+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 0x7fe5700769f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe57c034000 tx=0x7fe57c02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.155+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 msgr2=0x7fe58c06cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.155+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe58800c400 tx=0x7fe58800c8c0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 shutdown_connections 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5700744e0 0x7fe5700769f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe58c1f9b30 0x7fe58c1f9f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe58c0d2f10 0x7fe58c1421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 --2- 172.21.15.150:0/1474211554 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe58c0b2350 0x7fe58c06cdd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.156+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 >> 172.21.15.150:0/1474211554 conn(0x7fe58c0b76d0 msgr2=0x7fe58c0d2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:09.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.159+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 shutdown_connections 2024-04-25T14:02:09.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.161+0000 7fe594134640 1 -- 172.21.15.150:0/1474211554 wait complete. 2024-04-25T14:02:09.183 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-25T14:02:09.184 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-25T14:02:09.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.338+0000 7f0e0f496640 1 Processor -- start 2024-04-25T14:02:09.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.338+0000 7f0e0f496640 1 -- start start 2024-04-25T14:02:09.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.339+0000 7f0e0f496640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.339+0000 7f0e0f496640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e0806dbd0 0x7f0e080d50f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.339+0000 7f0e0f496640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e080d0e00 con 0x7f0e080d4ce0 2024-04-25T14:02:09.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.339+0000 7f0e0f496640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0e080d5630 con 0x7f0e0806dbd0 2024-04-25T14:02:09.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.339+0000 7f0e0f496640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e080d1630 con 0x7f0e081646b0 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e0806dbd0 0x7f0e080d50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0d232640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0ca31640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f0e080d4ce0 0x7f0e081603c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50610/0 (socket says 172.21.15.150:50610) 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0e0806dbd0 0x7f0e080d50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:51692/0 (socket says 172.21.15.150:51692) 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0d232640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:50644/0 (socket says 172.21.15.150:50644) 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0ca31640 1 -- 172.21.15.150:0/60321557 learned_addr learned my addr 172.21.15.150:0/60321557 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 -- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 msgr2=0x7f0e080d47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 -- 172.21.15.150:0/60321557 >> v1:172.21.15.187:6789/0 conn(0x7f0e080d4ce0 legacy=0x7f0e081603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0dfffff640 1 -- 172.21.15.150:0/60321557 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1177158530 0 0) 0x7f0e080d0e00 con 0x7f0e080d4ce0 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0da33640 1 -- 172.21.15.150:0/60321557 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e080d1070 con 0x7f0e0806dbd0 2024-04-25T14:02:09.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.340+0000 7f0e0d232640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:09.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.341+0000 7f0e0da33640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e080d50f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e000097a0 tx=0x7f0e0002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77fcd1ec8e82772f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:09.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.341+0000 7f0dfffff640 1 -- 172.21.15.150:0/60321557 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00037070 con 0x7f0e0806dbd0 2024-04-25T14:02:09.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.341+0000 7f0dfffff640 1 -- 172.21.15.150:0/60321557 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e0002ae50 con 0x7f0e0806dbd0 2024-04-25T14:02:09.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.341+0000 7f0dfffff640 1 -- 172.21.15.150:0/60321557 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00032660 con 0x7f0e0806dbd0 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 -- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 msgr2=0x7f0e080d50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e080d50f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0e000097a0 tx=0x7f0e0002a100 comp rx=0 tx=0).stop 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 -- 172.21.15.150:0/60321557 shutdown_connections 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e080d50f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 --2- 172.21.15.150:0/60321557 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e081646b0 0x7f0e080d47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 -- 172.21.15.150:0/60321557 >> 172.21.15.150:0/60321557 conn(0x7f0e080d2ce0 msgr2=0x7f0e0806ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 -- 172.21.15.150:0/60321557 shutdown_connections 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.342+0000 7f0e0f496640 1 -- 172.21.15.150:0/60321557 wait complete. 2024-04-25T14:02:09.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.344+0000 7f0e0f496640 1 Processor -- start 2024-04-25T14:02:09.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.344+0000 7f0e0f496640 1 -- start start 2024-04-25T14:02:09.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e08162060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e080d4ce0 0x7f0e08162c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0e08164ac0 con 0x7f0e081646b0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0806c730 con 0x7f0e0806dbd0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0f496640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0e08164c40 con 0x7f0e080d4ce0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36484/0 (socket says 172.21.15.150:36484) 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 -- 172.21.15.150:0/459038008 learned_addr learned my addr 172.21.15.150:0/459038008 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0d232640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e08162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0ca31640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e080d4ce0 0x7f0e08162c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 -- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e080d4ce0 msgr2=0x7f0e08162c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e080d4ce0 0x7f0e08162c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 -- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 msgr2=0x7f0e08162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e08162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.345+0000 7f0e0da33640 1 -- 172.21.15.150:0/459038008 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e081eb280 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0e0da33640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e00035040 tx=0x7f0e000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00037040 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e081eacc0 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e00004030 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e00004420 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.346+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e080ce210 con 0x7f0e081646b0 2024-04-25T14:02:09.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.347+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e08162970 con 0x7f0e081646b0 2024-04-25T14:02:09.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.348+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0e00003610 con 0x7f0e081646b0 2024-04-25T14:02:09.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.348+0000 7f0dfdffb640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 0x7f0dec0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:09.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.348+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f0e000b5e30 con 0x7f0e081646b0 2024-04-25T14:02:09.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.348+0000 7f0e0d232640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 0x7f0dec0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:09.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.349+0000 7f0e0d232640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 0x7f0dec0769f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009bd0 tx=0x7f0e0400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:09.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.353+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e081eb280 con 0x7f0e081646b0 2024-04-25T14:02:09.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:09.538+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f0e08076a40 con 0x7f0e081646b0 2024-04-25T14:02:10.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.157+0000 7f0dfdffb640 1 -- 172.21.15.150:0/459038008 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v20) ==== 157+0+0 (secure 0 0 0) 0x7f0e00080010 con 0x7f0e081646b0 2024-04-25T14:02:10.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 msgr2=0x7f0dec0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 0x7f0dec0769f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009bd0 tx=0x7f0e0400b040 comp rx=0 tx=0).stop 2024-04-25T14:02:10.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 msgr2=0x7f0e080ba530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0e00035040 tx=0x7f0e000046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 shutdown_connections 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0dec0744e0 0x7f0dec0769f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0e081646b0 0x7f0e080ba530 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0e080d4ce0 0x7f0e08162c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 --2- 172.21.15.150:0/459038008 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0e0806dbd0 0x7f0e08162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 >> 172.21.15.150:0/459038008 conn(0x7f0e080d2ce0 msgr2=0x7f0e0815c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 shutdown_connections 2024-04-25T14:02:10.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.161+0000 7f0e0f496640 1 -- 172.21.15.150:0/459038008 wait complete. 2024-04-25T14:02:10.173 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T14:02:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.289+0000 7fb9a5784640 1 Processor -- start 2024-04-25T14:02:10.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.289+0000 7fb9a5784640 1 -- start start 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb9a5784640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb9a00b4c10 0x7fb9a0160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb9a5784640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b91f0 0x7fb9a006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb9a5784640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9a00b9600 con 0x7fb9a006d630 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb9a5784640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb9a006cc60 con 0x7fb9a00b4c10 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb9a5784640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a00d4f20 con 0x7fb9a00b91f0 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb9a00b4c10 0x7fb9a0160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb9a006d630 0x7fb9a0163c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59558/0 (socket says 172.21.15.150:59558) 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99f7fe640 1 -- 172.21.15.150:0/1920037782 learned_addr learned my addr 172.21.15.150:0/1920037782 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99e7fc640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b91f0 0x7fb9a006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99effd640 1 -- 172.21.15.150:0/1920037782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b91f0 msgr2=0x7fb9a006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99effd640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b91f0 0x7fb9a006d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99effd640 1 -- 172.21.15.150:0/1920037782 >> v1:172.21.15.150:6789/0 conn(0x7fb9a006d630 legacy=0x7fb9a0163c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99effd640 1 -- 172.21.15.150:0/1920037782 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a00d5200 con 0x7fb9a00b4c10 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.291+0000 7fb99dffb640 1 -- 172.21.15.150:0/1920037782 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2643215834 0 0) 0x7fb9a00b9600 con 0x7fb9a006d630 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.292+0000 7fb99effd640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a00b4c10 0x7fb9a0160150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb990009d90 tx=0x7fb99002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3971d7bfdc505bb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.292+0000 7fb99dffb640 1 -- 172.21.15.150:0/1920037782 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990007740 con 0x7fb9a00b4c10 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.292+0000 7fb99dffb640 1 -- 172.21.15.150:0/1920037782 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9900078e0 con 0x7fb9a00b4c10 2024-04-25T14:02:10.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.292+0000 7fb99dffb640 1 -- 172.21.15.150:0/1920037782 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990007cb0 con 0x7fb9a00b4c10 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 -- 172.21.15.150:0/1920037782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a00b4c10 msgr2=0x7fb9a0160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a00b4c10 0x7fb9a0160150 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb990009d90 tx=0x7fb99002a660 comp rx=0 tx=0).stop 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 -- 172.21.15.150:0/1920037782 shutdown_connections 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b91f0 0x7fb9a006d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 --2- 172.21.15.150:0/1920037782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a00b4c10 0x7fb9a0160150 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 -- 172.21.15.150:0/1920037782 >> 172.21.15.150:0/1920037782 conn(0x7fb9a00b7690 msgr2=0x7fb9a00b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:10.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 -- 172.21.15.150:0/1920037782 shutdown_connections 2024-04-25T14:02:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.293+0000 7fb9a5784640 1 -- 172.21.15.150:0/1920037782 wait complete. 2024-04-25T14:02:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.294+0000 7fb9a5784640 1 Processor -- start 2024-04-25T14:02:10.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.295+0000 7fb9a5784640 1 -- start start 2024-04-25T14:02:10.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b4c10 0x7fb9a00ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9a00b91f0 0x7fb9a014f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a006c8f0 con 0x7fb9a006d630 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a006c610 con 0x7fb9a00b91f0 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb9a006da40 con 0x7fb9a00b4c10 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36506/0 (socket says 172.21.15.150:36506) 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 -- 172.21.15.150:0/2349989370 learned_addr learned my addr 172.21.15.150:0/2349989370 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99e7fc640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b4c10 0x7fb9a00ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99f7fe640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9a00b91f0 0x7fb9a014f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 -- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b4c10 msgr2=0x7fb9a00ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b4c10 0x7fb9a00ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 -- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9a00b91f0 msgr2=0x7fb9a014f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9a00b91f0 0x7fb9a014f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 -- 172.21.15.150:0/2349989370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a014e5c0 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb99effd640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb990009eb0 tx=0x7fb990004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99003f070 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb990031c40 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a014e8a0 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.296+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990043410 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.297+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a014ee60 con 0x7fb9a006d630 2024-04-25T14:02:10.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.297+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a0076a40 con 0x7fb9a006d630 2024-04-25T14:02:10.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.298+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb9900435b0 con 0x7fb9a006d630 2024-04-25T14:02:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.298+0000 7fb9877fe640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 0x7fb98c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.299+0000 7fb99e7fc640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 0x7fb98c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.299+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 6007+0+0 (secure 0 0 0) 0x7fb9900b6240 con 0x7fb9a006d630 2024-04-25T14:02:10.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.299+0000 7fb99e7fc640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 0x7fb98c0769c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb994006fd0 tx=0x7fb994008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:10.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.303+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb99004a070 con 0x7fb9a006d630 2024-04-25T14:02:10.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:10.479+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb9a00d4030 con 0x7fb9a006d630 2024-04-25T14:02:12.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.168+0000 7fb9877fe640 1 -- 172.21.15.150:0/2349989370 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v22) ==== 228+0+0 (secure 0 0 0) 0x7fb990080820 con 0x7fb9a006d630 2024-04-25T14:02:12.169 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data_ec' created 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.172+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 msgr2=0x7fb98c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.172+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 0x7fb98c0769c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb994006fd0 tx=0x7fb994008040 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.172+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 msgr2=0x7fb9a00cfce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.172+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb990009eb0 tx=0x7fb990004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 shutdown_connections 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb98c0744b0 0x7fb98c0769c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb9a00b91f0 0x7fb9a014f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb9a00b4c10 0x7fb9a00ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 --2- 172.21.15.150:0/2349989370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb9a006d630 0x7fb9a00cfce0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 >> 172.21.15.150:0/2349989370 conn(0x7fb9a00b7690 msgr2=0x7fb9a006ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:12.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.173+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 shutdown_connections 2024-04-25T14:02:12.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.175+0000 7fb9a5784640 1 -- 172.21.15.150:0/2349989370 wait complete. 2024-04-25T14:02:12.195 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-25T14:02:12.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.312+0000 7f566d651640 1 Processor -- start 2024-04-25T14:02:12.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.312+0000 7f566d651640 1 -- start start 2024-04-25T14:02:12.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f566d651640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680b5030 0x7f56680d0da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f566d651640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56680d4e40 0x7f566815f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f566d651640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56680d0b80 con 0x7f566806dc70 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f566d651640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f566806d7e0 con 0x7f56680d4e40 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f566d651640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f56680d14a0 con 0x7f56680b5030 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f56677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56680d4e40 0x7f566815f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f56677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56680d4e40 0x7f566815f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36510/0 (socket says 172.21.15.150:36510) 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f56677fe640 1 -- 172.21.15.150:0/1756518022 learned_addr learned my addr 172.21.15.150:0/1756518022 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:12.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.314+0000 7f565e7fc640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680b5030 0x7f56680d0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56677fe640 1 -- 172.21.15.150:0/1756518022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680b5030 msgr2=0x7f56680d0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:12.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56677fe640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680b5030 0x7f56680d0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56677fe640 1 -- 172.21.15.150:0/1756518022 >> v1:172.21.15.187:6789/0 conn(0x7f566806dc70 legacy=0x7f56680d4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56677fe640 1 -- 172.21.15.150:0/1756518022 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f566806d190 con 0x7f56680d4e40 2024-04-25T14:02:12.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56677fe640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56680d4e40 0x7f566815f760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5660009e50 tx=0x7f566002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e121529aa0eaa939 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:12.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56667fc640 1 -- 172.21.15.150:0/1756518022 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660036070 con 0x7f56680d4e40 2024-04-25T14:02:12.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56667fc640 1 -- 172.21.15.150:0/1756518022 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5660031600 con 0x7f56680d4e40 2024-04-25T14:02:12.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.315+0000 7f56667fc640 1 -- 172.21.15.150:0/1756518022 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56600319d0 con 0x7f56680d4e40 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 -- 172.21.15.150:0/1756518022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56680d4e40 msgr2=0x7f566815f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56680d4e40 0x7f566815f760 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5660009e50 tx=0x7f566002a710 comp rx=0 tx=0).stop 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 -- 172.21.15.150:0/1756518022 shutdown_connections 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56680d4e40 0x7f566815f760 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 --2- 172.21.15.150:0/1756518022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680b5030 0x7f56680d0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 -- 172.21.15.150:0/1756518022 >> 172.21.15.150:0/1756518022 conn(0x7f56680d2bc0 msgr2=0x7f56680d5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 -- 172.21.15.150:0/1756518022 shutdown_connections 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.316+0000 7f566d651640 1 -- 172.21.15.150:0/1756518022 wait complete. 2024-04-25T14:02:12.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.318+0000 7f566d651640 1 Processor -- start 2024-04-25T14:02:12.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.318+0000 7f566d651640 1 -- start start 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56680b5030 0x7f56680cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680d4e40 0x7f56681f9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f56680d53d0 con 0x7f566806dc70 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f56680d5250 con 0x7f56680b5030 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f566d651640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f56680d5550 con 0x7f56680d4e40 2024-04-25T14:02:12.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f5666ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f5666ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36522/0 (socket says 172.21.15.150:36522) 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.319+0000 7f5666ffd640 1 -- 172.21.15.150:0/3248843501 learned_addr learned my addr 172.21.15.150:0/3248843501 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f565e7fc640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56680b5030 0x7f56680cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f56677fe640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680d4e40 0x7f56681f9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 -- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680d4e40 msgr2=0x7f56681f9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680d4e40 0x7f56681f9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 -- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56680b5030 msgr2=0x7f56680cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56680b5030 0x7f56680cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 -- 172.21.15.150:0/3248843501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56681f7680 con 0x7f566806dc70 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f5666ffd640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f565800c440 tx=0x7f565800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56580052d0 con 0x7f566806dc70 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5658005470 con 0x7f566806dc70 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5658007260 con 0x7f566806dc70 2024-04-25T14:02:12.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56680ccb00 con 0x7f566806dc70 2024-04-25T14:02:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.320+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56680c9870 con 0x7f566806dc70 2024-04-25T14:02:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.321+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5668076a40 con 0x7f566806dc70 2024-04-25T14:02:12.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.322+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f56580074c0 con 0x7f566806dc70 2024-04-25T14:02:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.322+0000 7f565ffff640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 0x7f56480769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.322+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 6500+0+0 (secure 0 0 0) 0x7f5658094a20 con 0x7f566806dc70 2024-04-25T14:02:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.322+0000 7f565e7fc640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 0x7f56480769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:12.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.323+0000 7f565e7fc640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 0x7f56480769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5650006fd0 tx=0x7f5650008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:12.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.327+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f565805ee90 con 0x7f566806dc70 2024-04-25T14:02:12.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:12.505+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f5668069c60 con 0x7f566806dc70 2024-04-25T14:02:13.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.173+0000 7f565ffff640 1 -- 172.21.15.150:0/3248843501 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v23) ==== 169+0+0 (secure 0 0 0) 0x7f565805e790 con 0x7f566806dc70 2024-04-25T14:02:13.174 INFO:teuthology.orchestra.run.smithi150.stderr:set pool 4 allow_ec_overwrites to true 2024-04-25T14:02:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 msgr2=0x7f56480769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 0x7f56480769c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5650006fd0 tx=0x7f5650008040 comp rx=0 tx=0).stop 2024-04-25T14:02:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 msgr2=0x7f56680b3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f565800c440 tx=0x7f565800c900 comp rx=0 tx=0).stop 2024-04-25T14:02:13.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 shutdown_connections 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56480744b0 0x7f56480769c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56680d4e40 0x7f56681f9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56680b5030 0x7f56680cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 --2- 172.21.15.150:0/3248843501 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f566806dc70 0x7f56680b3de0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 >> 172.21.15.150:0/3248843501 conn(0x7f56680d2bc0 msgr2=0x7f566815d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 shutdown_connections 2024-04-25T14:02:13.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.176+0000 7f566d651640 1 -- 172.21.15.150:0/3248843501 wait complete. 2024-04-25T14:02:13.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-25T14:02:13.298 INFO:tasks.ceph.mds.b.smithi187.stderr:2024-04-25T14:02:13.299+0000 7f5757ae9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:02:13.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.316+0000 7f4dac28c640 1 Processor -- start 2024-04-25T14:02:13.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.316+0000 7f4dac28c640 1 -- start start 2024-04-25T14:02:13.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4dac28c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 0x7f4da40b4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4dac28c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4da40d07c0 0x7f4da40d0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4dac28c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4da40d1110 con 0x7f4da4163c50 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4dac28c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4da40d54c0 con 0x7f4da40d07c0 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4dac28c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4da40d48b0 con 0x7f4da40d13a0 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9827640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4da40d07c0 0x7f4da40d0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4daa829640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4da4163c50 0x7f4da4164060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59560/0 (socket says 172.21.15.150:59560) 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4daa829640 1 -- 172.21.15.150:0/2708266540 learned_addr learned my addr 172.21.15.150:0/2708266540 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4daa028640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 0x7f4da40b4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9827640 1 -- 172.21.15.150:0/2708266540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 msgr2=0x7f4da40b4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9827640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 0x7f4da40b4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9827640 1 -- 172.21.15.150:0/2708266540 >> v1:172.21.15.150:6789/0 conn(0x7f4da4163c50 legacy=0x7f4da4164060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9026640 1 -- 172.21.15.150:0/2708266540 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 270854524 0 0) 0x7f4d9c001110 con 0x7f4da4163c50 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.318+0000 7f4da9827640 1 -- 172.21.15.150:0/2708266540 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da40d4b90 con 0x7f4da40d07c0 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.319+0000 7f4daa028640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 0x7f4da40b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.319+0000 7f4da9827640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da40d07c0 0x7f4da40d0bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d940097a0 tx=0x7f4d9402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2f0796547a4cc94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.319+0000 7f4da9026640 1 -- 172.21.15.150:0/2708266540 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d94037070 con 0x7f4da40d07c0 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.319+0000 7f4da9026640 1 -- 172.21.15.150:0/2708266540 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d94032440 con 0x7f4da40d07c0 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.319+0000 7f4da9026640 1 -- 172.21.15.150:0/2708266540 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d94032810 con 0x7f4da40d07c0 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 -- 172.21.15.150:0/2708266540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da40d07c0 msgr2=0x7f4da40d0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da40d07c0 0x7f4da40d0bd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4d940097a0 tx=0x7f4d9402a180 comp rx=0 tx=0).stop 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 -- 172.21.15.150:0/2708266540 shutdown_connections 2024-04-25T14:02:13.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da40d07c0 0x7f4da40d0bd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 --2- 172.21.15.150:0/2708266540 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d13a0 0x7f4da40b4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 -- 172.21.15.150:0/2708266540 >> 172.21.15.150:0/2708266540 conn(0x7f4da40d6500 msgr2=0x7f4da40b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:13.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 -- 172.21.15.150:0/2708266540 shutdown_connections 2024-04-25T14:02:13.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.320+0000 7f4dac28c640 1 -- 172.21.15.150:0/2708266540 wait complete. 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.322+0000 7f4dac28c640 1 Processor -- start 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.322+0000 7f4dac28c640 1 -- start start 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d07c0 0x7f4da4148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40d13a0 0x7f4da41eba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40d4440 con 0x7f4da4163c50 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4da40d5150 con 0x7f4da40d13a0 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4dac28c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4da40d45c0 con 0x7f4da40d07c0 2024-04-25T14:02:13.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4daa829640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4daa829640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36536/0 (socket says 172.21.15.150:36536) 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4daa829640 1 -- 172.21.15.150:0/4226791856 learned_addr learned my addr 172.21.15.150:0/4226791856 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.323+0000 7f4daa028640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d07c0 0x7f4da4148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4da9827640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40d13a0 0x7f4da41eba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 -- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d07c0 msgr2=0x7f4da4148d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d07c0 0x7f4da4148d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 -- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40d13a0 msgr2=0x7f4da41eba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40d13a0 0x7f4da41eba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 -- 172.21.15.150:0/4226791856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da415d9e0 con 0x7f4da4163c50 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4daa829640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c410 tx=0x7f4d9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c017070 con 0x7f4da4163c50 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d9c005ce0 con 0x7f4da4163c50 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da41eadc0 con 0x7f4da4163c50 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0072f0 con 0x7f4da4163c50 2024-04-25T14:02:13.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.324+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da41496a0 con 0x7f4da4163c50 2024-04-25T14:02:13.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.325+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4da4076a40 con 0x7f4da4163c50 2024-04-25T14:02:13.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.326+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4d9c01a070 con 0x7f4da4163c50 2024-04-25T14:02:13.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.326+0000 7f4d9affd640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 0x7f4d880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:13.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.326+0000 7f4daa028640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 0x7f4d880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:13.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.326+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f4d9c0947b0 con 0x7f4da4163c50 2024-04-25T14:02:13.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.327+0000 7f4daa028640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 0x7f4d880769c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4da40d22c0 tx=0x7f4da00077c0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:13.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.331+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4da40d1110 con 0x7f4da4163c50 2024-04-25T14:02:13.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:13.545+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f4da40d3f60 con 0x7f4da4163c50 2024-04-25T14:02:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.175+0000 7f4d9affd640 1 -- 172.21.15.150:0/4226791856 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f4d9c05e660 con 0x7f4da4163c50 2024-04-25T14:02:14.175 INFO:teuthology.orchestra.run.smithi150.stderr:added data pool 4 to fsmap 2024-04-25T14:02:14.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 msgr2=0x7f4d880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 0x7f4d880769c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4da40d22c0 tx=0x7f4da00077c0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 msgr2=0x7f4da40c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00c410 tx=0x7f4d9c00c8d0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 shutdown_connections 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4d880744b0 0x7f4d880769c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4da4163c50 0x7f4da40c9620 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4da40d13a0 0x7f4da41eba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 --2- 172.21.15.150:0/4226791856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4da40d07c0 0x7f4da4148d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.178+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 >> 172.21.15.150:0/4226791856 conn(0x7f4da40d6500 msgr2=0x7f4da40ba700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.179+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 shutdown_connections 2024-04-25T14:02:14.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.179+0000 7f4dac28c640 1 -- 172.21.15.150:0/4226791856 wait complete. 2024-04-25T14:02:14.192 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.301+0000 7f637c37d640 1 Processor -- start 2024-04-25T14:02:14.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.302+0000 7f637c37d640 1 -- start start 2024-04-25T14:02:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637c37d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 0x7f63740d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637c37d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63740b9070 0x7f637406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637c37d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f637406c570 con 0x7f637406d830 2024-04-25T14:02:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637c37d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f63741605e0 con 0x7f63740b9070 2024-04-25T14:02:14.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637c37d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63740d5980 con 0x7f63741601d0 2024-04-25T14:02:14.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f637a119640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 0x7f63740d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f6379918640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63740b9070 0x7f637406d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f6379918640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f63740b9070 0x7f637406d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:51744/0 (socket says 172.21.15.150:51744) 2024-04-25T14:02:14.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.303+0000 7f6379918640 1 -- 172.21.15.150:0/1455071686 learned_addr learned my addr 172.21.15.150:0/1455071686 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:14.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f6379918640 1 -- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 msgr2=0x7f63740d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f6379918640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 0x7f63740d5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f6379918640 1 -- 172.21.15.150:0/1455071686 >> v1:172.21.15.187:6789/0 conn(0x7f637406d830 legacy=0x7f63740d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f6379918640 1 -- 172.21.15.150:0/1455071686 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f637406c810 con 0x7f63740b9070 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f637a119640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 0x7f63740d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.304+0000 7f6379918640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740b9070 0x7f637406d2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6364009870 tx=0x7f636402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9062aa7639c207c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f6379117640 1 -- 172.21.15.150:0/1455071686 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364037070 con 0x7f63740b9070 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f6379117640 1 -- 172.21.15.150:0/1455071686 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6364032440 con 0x7f63740b9070 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f6379117640 1 -- 172.21.15.150:0/1455071686 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6364032810 con 0x7f63740b9070 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 -- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740b9070 msgr2=0x7f637406d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740b9070 0x7f637406d2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6364009870 tx=0x7f636402a1c0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 -- 172.21.15.150:0/1455071686 shutdown_connections 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f63740b9070 0x7f637406d2f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 --2- 172.21.15.150:0/1455071686 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63741601d0 0x7f63740d5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 -- 172.21.15.150:0/1455071686 >> 172.21.15.150:0/1455071686 conn(0x7f63740d5d90 msgr2=0x7f63740d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 -- 172.21.15.150:0/1455071686 shutdown_connections 2024-04-25T14:02:14.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.305+0000 7f637c37d640 1 -- 172.21.15.150:0/1455071686 wait complete. 2024-04-25T14:02:14.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.307+0000 7f637c37d640 1 Processor -- start 2024-04-25T14:02:14.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.307+0000 7f637c37d640 1 -- start start 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f637406d830 0x7f63741573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 0x7f637415de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f63740d5640 con 0x7f63741601d0 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f637406cdd0 con 0x7f637406d830 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637c37d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f63740b9480 con 0x7f63740b9070 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f637a91a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.308+0000 7f6379918640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 0x7f637415de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36546/0 (socket says 172.21.15.150:36546) 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 -- 172.21.15.150:0/3973025692 learned_addr learned my addr 172.21.15.150:0/3973025692 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:14.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a119640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f637406d830 0x7f63741573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 -- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 msgr2=0x7f637415de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 0x7f637415de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 -- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f637406d830 msgr2=0x7f63741573d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f637406d830 0x7f63741573d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 -- 172.21.15.150:0/3973025692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63740cedb0 con 0x7f63741601d0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f6379918640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 0x7f637415de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637a91a640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c3f0 tx=0x7f636c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c017070 con 0x7f63741601d0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f636c0052d0 con 0x7f63741601d0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f636c003850 con 0x7f63741601d0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63740ccf20 con 0x7f63741601d0 2024-04-25T14:02:14.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.309+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63740cd1c0 con 0x7f63741601d0 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.310+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6374076a40 con 0x7f63741601d0 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.311+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f636c01a070 con 0x7f63741601d0 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.311+0000 7f636affd640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 0x7f63580769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.311+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f636c094620 con 0x7f63741601d0 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.311+0000 7f637a119640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 0x7f63580769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.312+0000 7f637a119640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 0x7f63580769f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6370009bd0 tx=0x7f637000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.316+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f636c05e540 con 0x7f63741601d0 2024-04-25T14:02:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.529+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f63740c9660 con 0x7f63741601d0 2024-04-25T14:02:14.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.529+0000 7f636affd640 1 -- 172.21.15.150:0/3973025692 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5186 (secure 0 0 0) 0x7f636c062840 con 0x7f63741601d0 2024-04-25T14:02:14.530 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:14.530 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:13.547854+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 6 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 msgr2=0x7f63580769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 0x7f63580769f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6370009bd0 tx=0x7f637000b040 comp rx=0 tx=0).stop 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 msgr2=0x7f6374162990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f636c00c3f0 tx=0x7f636c00c8b0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 shutdown_connections 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63580744e0 0x7f63580769f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63741601d0 0x7f6374162990 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63740b9070 0x7f637415de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 --2- 172.21.15.150:0/3973025692 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f637406d830 0x7f63741573d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 >> 172.21.15.150:0/3973025692 conn(0x7f63740d5d90 msgr2=0x7f63740b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 shutdown_connections 2024-04-25T14:02:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.531+0000 7f637c37d640 1 -- 172.21.15.150:0/3973025692 wait complete. 2024-04-25T14:02:14.542 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:14.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.651+0000 7f3e579ce640 1 Processor -- start 2024-04-25T14:02:14.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.651+0000 7f3e579ce640 1 -- start start 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e579ce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e50158760 0x7f3e501625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e579ce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e5015e520 0x7f3e5015e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e579ce640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e5015ee70 con 0x7f3e500b5400 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e579ce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3e50070fd0 con 0x7f3e5015e520 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e579ce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5015f100 con 0x7f3e50158760 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e55f6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e5015e520 0x7f3e5015e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e55f6b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3e5015e520 0x7f3e5015e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36552/0 (socket says 172.21.15.150:36552) 2024-04-25T14:02:14.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.652+0000 7f3e55f6b640 1 -- 172.21.15.150:0/2972136592 learned_addr learned my addr 172.21.15.150:0/2972136592 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:14.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e5576a640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e50158760 0x7f3e501625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e55f6b640 1 -- 172.21.15.150:0/2972136592 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e50158760 msgr2=0x7f3e501625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:14.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e55f6b640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e50158760 0x7f3e501625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e55f6b640 1 -- 172.21.15.150:0/2972136592 >> v1:172.21.15.187:6789/0 conn(0x7f3e500b5400 legacy=0x7f3e500b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e55f6b640 1 -- 172.21.15.150:0/2972136592 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e5015f3e0 con 0x7f3e5015e520 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.653+0000 7f3e55f6b640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e5015e520 0x7f3e5015e930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e50 tx=0x7f3e4802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1228909001b3cde7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e47fff640 1 -- 172.21.15.150:0/2972136592 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e48036070 con 0x7f3e5015e520 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e47fff640 1 -- 172.21.15.150:0/2972136592 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e48031600 con 0x7f3e5015e520 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e47fff640 1 -- 172.21.15.150:0/2972136592 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e480319d0 con 0x7f3e5015e520 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 -- 172.21.15.150:0/2972136592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e5015e520 msgr2=0x7f3e5015e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e5015e520 0x7f3e5015e930 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3e48009e50 tx=0x7f3e4802a710 comp rx=0 tx=0).stop 2024-04-25T14:02:14.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 -- 172.21.15.150:0/2972136592 shutdown_connections 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e5015e520 0x7f3e5015e930 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 --2- 172.21.15.150:0/2972136592 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e50158760 0x7f3e501625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 -- 172.21.15.150:0/2972136592 >> 172.21.15.150:0/2972136592 conn(0x7f3e500d0400 msgr2=0x7f3e500d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 -- 172.21.15.150:0/2972136592 shutdown_connections 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.654+0000 7f3e579ce640 1 -- 172.21.15.150:0/2972136592 wait complete. 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.656+0000 7f3e579ce640 1 Processor -- start 2024-04-25T14:02:14.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.656+0000 7f3e579ce640 1 -- start start 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e500b5400 0x7f3e5014c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5015e520 0x7f3e501f7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501620d0 con 0x7f3e50158760 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50161df0 con 0x7f3e5015e520 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.657+0000 7f3e579ce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3e500703d0 con 0x7f3e500b5400 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36564/0 (socket says 172.21.15.150:36564) 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 -- 172.21.15.150:0/297521304 learned_addr learned my addr 172.21.15.150:0/297521304 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e5576a640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e500b5400 0x7f3e5014c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e55f6b640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5015e520 0x7f3e501f7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 -- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e500b5400 msgr2=0x7f3e5014c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e500b5400 0x7f3e5014c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 -- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5015e520 msgr2=0x7f3e501f7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5015e520 0x7f3e501f7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 -- 172.21.15.150:0/297521304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e501f48b0 con 0x7f3e50158760 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e54f69640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000c420 tx=0x7f3e4000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400052d0 con 0x7f3e50158760 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e40005490 con 0x7f3e50158760 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40007260 con 0x7f3e50158760 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e5014cf60 con 0x7f3e50158760 2024-04-25T14:02:14.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.658+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e50152ca0 con 0x7f3e50158760 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.660+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e50076a40 con 0x7f3e50158760 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.660+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3e40007400 con 0x7f3e50158760 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.661+0000 7f3e45ffb640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 0x7f3e340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.661+0000 7f3e5576a640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 0x7f3e340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.661+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3e40094870 con 0x7f3e50158760 2024-04-25T14:02:14.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.661+0000 7f3e5576a640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 0x7f3e340769f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e50074b90 tx=0x7f3e4c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.665+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3e4005e670 con 0x7f3e50158760 2024-04-25T14:02:14.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.841+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e500cf890 con 0x7f3e50158760 2024-04-25T14:02:14.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.842+0000 7f3e45ffb640 1 -- 172.21.15.150:0/297521304 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f3e40062970 con 0x7f3e50158760 2024-04-25T14:02:14.842 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:14.843 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 msgr2=0x7f3e340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 0x7f3e340769f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e50074b90 tx=0x7f3e4c008040 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 msgr2=0x7f3e501559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000c420 tx=0x7f3e4000c8e0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 shutdown_connections 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3e340744e0 0x7f3e340769f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3e5015e520 0x7f3e501f7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3e50158760 0x7f3e501559b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 --2- 172.21.15.150:0/297521304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3e500b5400 0x7f3e5014c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 >> 172.21.15.150:0/297521304 conn(0x7f3e500d0400 msgr2=0x7f3e5015ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 shutdown_connections 2024-04-25T14:02:14.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.844+0000 7f3e579ce640 1 -- 172.21.15.150:0/297521304 wait complete. 2024-04-25T14:02:14.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.962+0000 7fd685e05640 1 Processor -- start 2024-04-25T14:02:14.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.962+0000 7fd685e05640 1 -- start start 2024-04-25T14:02:14.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd685e05640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 0x7fd68006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd685e05640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 0x7fd6800ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd685e05640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6800b4750 con 0x7fd68006f780 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd685e05640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd68015e310 con 0x7fd6800b5f00 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd685e05640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6800b5360 con 0x7fd68015df00 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67effd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd68006f780 0x7fd6800b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59566/0 (socket says 172.21.15.150:59566) 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67effd640 1 -- 172.21.15.150:0/3251064734 learned_addr learned my addr 172.21.15.150:0/3251064734 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3251064734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 0x7fd68006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67ffff640 1 --2- 172.21.15.150:0/3251064734 >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 0x7fd6800ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67e7fc640 1 -- 172.21.15.150:0/3251064734 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602007309 0 0) 0x7fd6800b4750 con 0x7fd68006f780 2024-04-25T14:02:14.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67e7fc640 1 -- 172.21.15.150:0/3251064734 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd668003520 con 0x7fd68006f780 2024-04-25T14:02:14.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67f7fe640 1 -- 172.21.15.150:0/3251064734 >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 msgr2=0x7fd6800ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:14.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3251064734 >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 0x7fd6800ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67f7fe640 1 -- 172.21.15.150:0/3251064734 >> v1:172.21.15.150:6789/0 conn(0x7fd68006f780 legacy=0x7fd6800b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67f7fe640 1 -- 172.21.15.150:0/3251064734 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6800b4a30 con 0x7fd68015df00 2024-04-25T14:02:14.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.964+0000 7fd67ffff640 1 --2- 172.21.15.150:0/3251064734 >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 0x7fd6800ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:14.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.965+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3251064734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 0x7fd68006f240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4152ecb1511847c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.965+0000 7fd67e7fc640 1 -- 172.21.15.150:0/3251064734 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670037070 con 0x7fd68015df00 2024-04-25T14:02:14.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.965+0000 7fd67e7fc640 1 -- 172.21.15.150:0/3251064734 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67002ae50 con 0x7fd68015df00 2024-04-25T14:02:14.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.965+0000 7fd67e7fc640 1 -- 172.21.15.150:0/3251064734 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670032640 con 0x7fd68015df00 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 -- 172.21.15.150:0/3251064734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 msgr2=0x7fd68006f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 --2- 172.21.15.150:0/3251064734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 0x7fd68006f240 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd6700097a0 tx=0x7fd67002a100 comp rx=0 tx=0).stop 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 -- 172.21.15.150:0/3251064734 shutdown_connections 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 --2- 172.21.15.150:0/3251064734 >> v2:172.21.15.187:3300/0 conn(0x7fd6800b5f00 0x7fd6800ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 --2- 172.21.15.150:0/3251064734 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68015df00 0x7fd68006f240 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 -- 172.21.15.150:0/3251064734 >> 172.21.15.150:0/3251064734 conn(0x7fd6800d1ae0 msgr2=0x7fd6800d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 -- 172.21.15.150:0/3251064734 shutdown_connections 2024-04-25T14:02:14.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.966+0000 7fd685e05640 1 -- 172.21.15.150:0/3251064734 wait complete. 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.968+0000 7fd685e05640 1 Processor -- start 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.968+0000 7fd685e05640 1 -- start start 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68006f780 0x7fd68009bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6800b5f00 0x7fd680159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800aed20 con 0x7fd68015df00 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd6800b4ff0 con 0x7fd6800b5f00 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.969+0000 7fd685e05640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6800aeea0 con 0x7fd68006f780 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36568/0 (socket says 172.21.15.150:36568) 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 -- 172.21.15.150:0/3744601882 learned_addr learned my addr 172.21.15.150:0/3744601882 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68006f780 0x7fd68009bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67effd640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6800b5f00 0x7fd680159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 -- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68006f780 msgr2=0x7fd68009bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68006f780 0x7fd68009bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 -- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6800b5f00 msgr2=0x7fd680159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6800b5f00 0x7fd680159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 -- 172.21.15.150:0/3744601882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68009e3f0 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd67ffff640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd678009e00 tx=0x7fd67800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678018020 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd678014480 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68009e6d0 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678014870 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.970+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd68009ec90 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.971+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6800b6480 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.972+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd678005d10 con 0x7fd68015df00 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.972+0000 7fd667fff640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 0x7fd66c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.972+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 0x7fd66c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:14.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.972+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd6780962e0 con 0x7fd68015df00 2024-04-25T14:02:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.973+0000 7fd67f7fe640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 0x7fd66c0769c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd67002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:14.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:14.977+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd6800b6480 con 0x7fd68015df00 2024-04-25T14:02:15.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.152+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd680076a40 con 0x7fd68015df00 2024-04-25T14:02:15.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.153+0000 7fd667fff640 1 -- 172.21.15.150:0/3744601882 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fd678022030 con 0x7fd68015df00 2024-04-25T14:02:15.153 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:15.154 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:15.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 msgr2=0x7fd66c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 0x7fd66c0769c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd670034000 tx=0x7fd67002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 msgr2=0x7fd6800cddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd678009e00 tx=0x7fd67800c490 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 shutdown_connections 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd66c0744b0 0x7fd66c0769c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd68015df00 0x7fd6800cddc0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd6800b5f00 0x7fd680159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 --2- 172.21.15.150:0/3744601882 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd68006f780 0x7fd68009bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 >> 172.21.15.150:0/3744601882 conn(0x7fd6800d1ae0 msgr2=0x7fd68015c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 shutdown_connections 2024-04-25T14:02:15.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.155+0000 7fd685e05640 1 -- 172.21.15.150:0/3744601882 wait complete. 2024-04-25T14:02:15.183 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:15.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.291+0000 7fe52a185640 1 Processor -- start 2024-04-25T14:02:15.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.291+0000 7fe52a185640 1 -- start start 2024-04-25T14:02:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.292+0000 7fe52a185640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5240d2ef0 0x7fe5240b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.292+0000 7fe52a185640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.292+0000 7fe52a185640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe52406f880 con 0x7fe5240d1ef0 2024-04-25T14:02:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.292+0000 7fe52a185640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe5240b4ba0 con 0x7fe5240d2ef0 2024-04-25T14:02:15.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.292+0000 7fe52a185640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe52406f2e0 con 0x7fe5240cf550 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe523fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5240d2ef0 0x7fe5240b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe523fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:50756/0 (socket says 172.21.15.150:50756) 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe523fff640 1 -- 172.21.15.150:0/1666764994 learned_addr learned my addr 172.21.15.150:0/1666764994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 --2- 172.21.15.150:0/1666764994 >> v2:172.21.15.187:3300/0 conn(0x7fe5240d2ef0 0x7fe5240b21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:51780/0 (socket says 172.21.15.150:51780) 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe5237fe640 1 --1- 172.21.15.150:0/1666764994 >> v1:172.21.15.187:6789/0 conn(0x7fe5240d1ef0 0x7fe5240d27e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50640/0 (socket says 172.21.15.150:50640) 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 -- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 msgr2=0x7fe5240d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 -- 172.21.15.150:0/1666764994 >> v1:172.21.15.187:6789/0 conn(0x7fe5240d1ef0 legacy=0x7fe5240d27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe522ffd640 1 -- 172.21.15.150:0/1666764994 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52406f5a0 con 0x7fe5240d2ef0 2024-04-25T14:02:15.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.293+0000 7fe523fff640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:15.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.294+0000 7fe522ffd640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240d2ef0 0x7fe5240b21b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=316eff7d96d3c8c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.294+0000 7fe5227fc640 1 -- 172.21.15.150:0/1666764994 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518037070 con 0x7fe5240d2ef0 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.294+0000 7fe5227fc640 1 -- 172.21.15.150:0/1666764994 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51802ae50 con 0x7fe5240d2ef0 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.294+0000 7fe5227fc640 1 -- 172.21.15.150:0/1666764994 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518032660 con 0x7fe5240d2ef0 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 -- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240d2ef0 msgr2=0x7fe5240b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240d2ef0 0x7fe5240b21b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe5180097a0 tx=0x7fe51802a100 comp rx=0 tx=0).stop 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 -- 172.21.15.150:0/1666764994 shutdown_connections 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5240cf550 0x7fe5240d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 --2- 172.21.15.150:0/1666764994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240d2ef0 0x7fe5240b21b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 -- 172.21.15.150:0/1666764994 >> 172.21.15.150:0/1666764994 conn(0x7fe52409b4a0 msgr2=0x7fe52409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 -- 172.21.15.150:0/1666764994 shutdown_connections 2024-04-25T14:02:15.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.295+0000 7fe52a185640 1 -- 172.21.15.150:0/1666764994 wait complete. 2024-04-25T14:02:15.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.296+0000 7fe52a185640 1 Processor -- start 2024-04-25T14:02:15.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.297+0000 7fe52a185640 1 -- start start 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 0x7fe524152d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 0x7fe5240d1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe5240b5030 con 0x7fe5240d1ef0 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe5240cf960 con 0x7fe5240cf550 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe52a185640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe5240b51b0 con 0x7fe524156790 2024-04-25T14:02:15.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36572/0 (socket says 172.21.15.150:36572) 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 -- 172.21.15.150:0/2154171084 learned_addr learned my addr 172.21.15.150:0/2154171084 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe5237fe640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 0x7fe524152d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe523fff640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 0x7fe5240d1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 -- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 msgr2=0x7fe5240d1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 0x7fe5240d1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 -- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 msgr2=0x7fe524152d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 0x7fe524152d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 -- 172.21.15.150:0/2154171084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe52415e5c0 con 0x7fe5240d1ef0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe523fff640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 0x7fe5240d1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe5237fe640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 0x7fe524152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.298+0000 7fe522ffd640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe5180098c0 tx=0x7fe5180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.299+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518041030 con 0x7fe5240d1ef0 2024-04-25T14:02:15.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.299+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe51802ac50 con 0x7fe5240d1ef0 2024-04-25T14:02:15.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.299+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe524153de0 con 0x7fe5240d1ef0 2024-04-25T14:02:15.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.299+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518037040 con 0x7fe5240d1ef0 2024-04-25T14:02:15.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.299+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe524154420 con 0x7fe5240d1ef0 2024-04-25T14:02:15.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.300+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe524076a40 con 0x7fe5240d1ef0 2024-04-25T14:02:15.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.300+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe518044070 con 0x7fe5240d1ef0 2024-04-25T14:02:15.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.301+0000 7fe50bfff640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 0x7fe5100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.301+0000 7fe5237fe640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 0x7fe5100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.301+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe51802ae00 con 0x7fe5240d1ef0 2024-04-25T14:02:15.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.301+0000 7fe5237fe640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 0x7fe5100769c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514007ce0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.305+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe51807fcb0 con 0x7fe5240d1ef0 2024-04-25T14:02:15.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.520+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe5241f77e0 con 0x7fe5240d1ef0 2024-04-25T14:02:15.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.521+0000 7fe50bfff640 1 -- 172.21.15.150:0/2154171084 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5186 (secure 0 0 0) 0x7fe51807f5b0 con 0x7fe5240d1ef0 2024-04-25T14:02:15.521 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:15.521 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:13.547854+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:15.522 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 6 2024-04-25T14:02:15.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.522+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 msgr2=0x7fe5100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.522+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 0x7fe5100769c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe514000f60 tx=0x7fe514007ce0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.522+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 msgr2=0x7fe5240d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.522+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe5180098c0 tx=0x7fe5180046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 shutdown_connections 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5100744b0 0x7fe5100769c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe524156790 0x7fe5240d1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5240d1ef0 0x7fe5240d1040 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 --2- 172.21.15.150:0/2154171084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5240cf550 0x7fe524152d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 >> 172.21.15.150:0/2154171084 conn(0x7fe52409b4a0 msgr2=0x7fe5240ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 shutdown_connections 2024-04-25T14:02:15.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.523+0000 7fe52a185640 1 -- 172.21.15.150:0/2154171084 wait complete. 2024-04-25T14:02:15.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:15.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.640+0000 7f4acfde3640 1 Processor -- start 2024-04-25T14:02:15.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.640+0000 7f4acfde3640 1 -- start start 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acfde3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4ac80b4e00 0x7f4ac80d4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acfde3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ac806cf80 0x7f4ac80d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acfde3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ac80d14d0 con 0x7f4ac80d10c0 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acfde3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac80b9680 con 0x7f4ac806cf80 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acfde3640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4ac806d790 con 0x7f4ac80b4e00 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ac806cf80 0x7f4ac80d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4ace380640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4ac80d10c0 0x7f4ac80d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:59582/0 (socket says 172.21.15.150:59582) 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4ace380640 1 -- 172.21.15.150:0/1077059824 learned_addr learned my addr 172.21.15.150:0/1077059824 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4ac806cf80 0x7f4ac80d0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36578/0 (socket says 172.21.15.150:36578) 2024-04-25T14:02:15.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acdb7f640 1 --2- 172.21.15.150:0/1077059824 >> v2:172.21.15.187:3300/0 conn(0x7f4ac80b4e00 0x7f4ac80d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 -- 172.21.15.150:0/1077059824 >> v2:172.21.15.187:3300/0 conn(0x7f4ac80b4e00 msgr2=0x7f4ac80d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 --2- 172.21.15.150:0/1077059824 >> v2:172.21.15.187:3300/0 conn(0x7f4ac80b4e00 0x7f4ac80d4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 -- 172.21.15.150:0/1077059824 >> v1:172.21.15.150:6789/0 conn(0x7f4ac80d10c0 legacy=0x7f4ac80d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4acd37e640 1 -- 172.21.15.150:0/1077059824 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac806da70 con 0x7f4ac806cf80 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.642+0000 7f4accb7d640 1 -- 172.21.15.150:0/1077059824 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 588323831 0 0) 0x7f4ac0001110 con 0x7f4ac80d10c0 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.643+0000 7f4acd37e640 1 --2- 172.21.15.150:0/1077059824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac806cf80 0x7f4ac80d0b80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4ac808eaf0 tx=0x7f4ab802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a54f27cd4a4ce44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.643+0000 7f4accb7d640 1 -- 172.21.15.150:0/1077059824 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8037070 con 0x7f4ac806cf80 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.643+0000 7f4accb7d640 1 -- 172.21.15.150:0/1077059824 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab8032440 con 0x7f4ac806cf80 2024-04-25T14:02:15.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.643+0000 7f4accb7d640 1 -- 172.21.15.150:0/1077059824 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8032810 con 0x7f4ac806cf80 2024-04-25T14:02:15.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 -- 172.21.15.150:0/1077059824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac806cf80 msgr2=0x7f4ac80d0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1077059824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac806cf80 0x7f4ac80d0b80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f4ac808eaf0 tx=0x7f4ab802a230 comp rx=0 tx=0).stop 2024-04-25T14:02:15.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 -- 172.21.15.150:0/1077059824 shutdown_connections 2024-04-25T14:02:15.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1077059824 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac806cf80 0x7f4ac80d0b80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1077059824 >> v2:172.21.15.187:3300/0 conn(0x7f4ac80b4e00 0x7f4ac80d4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 -- 172.21.15.150:0/1077059824 >> 172.21.15.150:0/1077059824 conn(0x7f4ac80d63d0 msgr2=0x7f4ac80d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 -- 172.21.15.150:0/1077059824 shutdown_connections 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.644+0000 7f4acfde3640 1 -- 172.21.15.150:0/1077059824 wait complete. 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.645+0000 7f4acfde3640 1 Processor -- start 2024-04-25T14:02:15.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.646+0000 7f4acfde3640 1 -- start start 2024-04-25T14:02:15.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ac806cf80 0x7f4ac8151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ac80d10c0 0x7f4ac81f5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac806d390 con 0x7f4ac80b4e00 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac80b92a0 con 0x7f4ac80d10c0 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acfde3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4ac806d510 con 0x7f4ac806cf80 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36584/0 (socket says 172.21.15.150:36584) 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 -- 172.21.15.150:0/1200055040 learned_addr learned my addr 172.21.15.150:0/1200055040 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acdb7f640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ac806cf80 0x7f4ac8151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4ace380640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ac80d10c0 0x7f4ac81f5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 -- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ac806cf80 msgr2=0x7f4ac8151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ac806cf80 0x7f4ac8151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 -- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ac80d10c0 msgr2=0x7f4ac81f5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ac80d10c0 0x7f4ac81f5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 -- 172.21.15.150:0/1200055040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac80cb0a0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4acd37e640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009890 tx=0x7f4ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.647+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8037040 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.648+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ab8042070 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.648+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8030920 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.648+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac80cb380 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.648+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac81525d0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.649+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac80cae10 con 0x7f4ac80b4e00 2024-04-25T14:02:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.649+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4ab8030ac0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.650+0000 7f4abe7fc640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 0x7f4aac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.650+0000 7f4acdb7f640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 0x7f4aac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.650+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f4ab80b5df0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.650+0000 7f4acdb7f640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 0x7f4aac0769f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40073f0 tx=0x7f4ac400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:15.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.654+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ab807fbf0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.832+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ac8076a40 con 0x7f4ac80b4e00 2024-04-25T14:02:15.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.834+0000 7f4abe7fc640 1 -- 172.21.15.150:0/1200055040 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f4ab8083ef0 con 0x7f4ac80b4e00 2024-04-25T14:02:15.834 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:15.834 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:15.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 msgr2=0x7f4aac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 0x7f4aac0769f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40073f0 tx=0x7f4ac400b040 comp rx=0 tx=0).stop 2024-04-25T14:02:15.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 msgr2=0x7f4ac8151cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:15.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009890 tx=0x7f4ab80046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 shutdown_connections 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4aac0744e0 0x7f4aac0769f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4ac80d10c0 0x7f4ac81f5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4ac80b4e00 0x7f4ac8151cd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 --2- 172.21.15.150:0/1200055040 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4ac806cf80 0x7f4ac8151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 >> 172.21.15.150:0/1200055040 conn(0x7f4ac80d63d0 msgr2=0x7f4ac80b6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 shutdown_connections 2024-04-25T14:02:15.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:15.836+0000 7f4acfde3640 1 -- 172.21.15.150:0/1200055040 wait complete. 2024-04-25T14:02:16.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:02:08.490370+0000', 'modified': '2024-04-25T14:02:13.547854+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3860691734', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 3860691734}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3860691734}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-04-25T14:02:16.250 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:02:16.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T14:02:16.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.359+0000 7fe2f1b6c640 1 Processor -- start 2024-04-25T14:02:16.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.359+0000 7fe2f1b6c640 1 -- start start 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2f1b6c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 0x7fe2ec06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2f1b6c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2ec163d80 0x7fe2ec0b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2f1b6c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec06ff50 con 0x7fe2ec0b5900 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2f1b6c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe2ec0b5d10 con 0x7fe2ec163d80 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2f1b6c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ec06f3d0 con 0x7fe2ec160460 2024-04-25T14:02:16.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.360+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 0x7fe2ec06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2ec163d80 0x7fe2ec0b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 0x7fe2ec06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:50772/0 (socket says 172.21.15.150:50772) 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/2974986935 learned_addr learned my addr 172.21.15.150:0/2974986935 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe2ec163d80 0x7fe2ec0b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:51824/0 (socket says 172.21.15.150:51824) 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2ebfff640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe2ec0b5900 0x7fe2ec072f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50646/0 (socket says 172.21.15.150:50646) 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 -- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 msgr2=0x7fe2ec06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 0x7fe2ec06ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 -- 172.21.15.150:0/2974986935 >> v1:172.21.15.187:6789/0 conn(0x7fe2ec0b5900 legacy=0x7fe2ec072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.361+0000 7fe2eaffd640 1 -- 172.21.15.150:0/2974986935 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec06f6b0 con 0x7fe2ec163d80 2024-04-25T14:02:16.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec163d80 0x7fe2ec0b53c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097a0 tx=0x7fe2e002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0fbcc65026386ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2ea7fc640 1 -- 172.21.15.150:0/2974986935 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0037070 con 0x7fe2ec163d80 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2ea7fc640 1 -- 172.21.15.150:0/2974986935 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e002ae50 con 0x7fe2ec163d80 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2ea7fc640 1 -- 172.21.15.150:0/2974986935 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0032660 con 0x7fe2ec163d80 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec163d80 msgr2=0x7fe2ec0b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec163d80 0x7fe2ec0b53c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097a0 tx=0x7fe2e002a100 comp rx=0 tx=0).stop 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/2974986935 shutdown_connections 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec163d80 0x7fe2ec0b53c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.362+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/2974986935 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec160460 0x7fe2ec06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.363+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/2974986935 >> 172.21.15.150:0/2974986935 conn(0x7fe2ec0d03c0 msgr2=0x7fe2ec0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.363+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/2974986935 shutdown_connections 2024-04-25T14:02:16.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.363+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/2974986935 wait complete. 2024-04-25T14:02:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.364+0000 7fe2f1b6c640 1 Processor -- start 2024-04-25T14:02:16.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.364+0000 7fe2f1b6c640 1 -- start start 2024-04-25T14:02:16.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 0x7fe2ec0b78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 0x7fe2ec14cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec0b3030 con 0x7fe2ec0cd120 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec06fc70 con 0x7fe2ec0b5900 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2f1b6c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ec0ce5c0 con 0x7fe2ec163d80 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36590/0 (socket says 172.21.15.150:36590) 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 -- 172.21.15.150:0/601023568 learned_addr learned my addr 172.21.15.150:0/601023568 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 0x7fe2ec0b78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 0x7fe2ec14cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 -- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 msgr2=0x7fe2ec14cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 0x7fe2ec14cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 -- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 msgr2=0x7fe2ec0b78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 0x7fe2ec0b78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 -- 172.21.15.150:0/601023568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec14d290 con 0x7fe2ec0cd120 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 0x7fe2ec14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 0x7fe2ec0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.366+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c730 tx=0x7fe2e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.367+0000 7fe2d3fff640 1 -- 172.21.15.150:0/601023568 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e4018070 con 0x7fe2ec0cd120 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.367+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ec0b6840 con 0x7fe2ec0cd120 2024-04-25T14:02:16.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.367+0000 7fe2d3fff640 1 -- 172.21.15.150:0/601023568 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2e4004d60 con 0x7fe2ec0cd120 2024-04-25T14:02:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.367+0000 7fe2d3fff640 1 -- 172.21.15.150:0/601023568 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400a860 con 0x7fe2ec0cd120 2024-04-25T14:02:16.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.367+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2ec0b6e00 con 0x7fe2ec0cd120 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.368+0000 7fe2d3fff640 1 -- 172.21.15.150:0/601023568 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe2e400ac10 con 0x7fe2ec0cd120 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.369+0000 7fe2d3fff640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 0x7fe2d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.369+0000 7fe2d3fff640 1 -- 172.21.15.150:0/601023568 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2e4094ca0 con 0x7fe2ec0cd120 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.369+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 0x7fe2d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.369+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 0x7fe2d80769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06e7e0 tx=0x7fe2dc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.370+0000 7fe2f8a17740 1 Processor -- start 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.370+0000 7fe2f8a17740 1 -- start start 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ec3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ada590 0x55d3d2e9d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2e01740 con 0x55d3d2afb270 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2d9d3a0 con 0x55d3d2ad23b0 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55d3d2dfd700 con 0x55d3d2ada590 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36604/0 (socket says 172.21.15.150:36604) 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/960964071 learned_addr learned my addr 172.21.15.150:0/960964071 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ec3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ada590 0x55d3d2e9d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ada590 msgr2=0x55d3d2e9d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ada590 0x55d3d2e9d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 msgr2=0x55d3d2ec3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ec3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.371+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/960964071 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d2db9750 con 0x55d3d2afb270 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc011d80 tx=0x7fe2dc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eb4a69231763388 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2d17fa640 1 -- 172.21.15.150:0/960964071 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc011db0 con 0x55d3d2afb270 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2d17fa640 1 -- 172.21.15.150:0/960964071 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc026070 con 0x55d3d2afb270 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2d17fa640 1 -- 172.21.15.150:0/960964071 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc021660 con 0x55d3d2afb270 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2f8a17740 1 -- 172.21.15.150:0/960964071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 msgr2=0x55d3d2ec7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.372+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc011d80 tx=0x7fe2dc014000 comp rx=0 tx=0).stop 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 -- 172.21.15.150:0/960964071 shutdown_connections 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ada590 0x55d3d2e9d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ec3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/960964071 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2afb270 0x55d3d2ec7600 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 -- 172.21.15.150:0/960964071 >> 172.21.15.150:0/960964071 conn(0x55d3d2aef3c0 msgr2=0x55d3d2e9c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 -- 172.21.15.150:0/960964071 shutdown_connections 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.373+0000 7fe2f8a17740 1 -- 172.21.15.150:0/960964071 wait complete. 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.374+0000 7fe2f8a17740 1 -- start start 2024-04-25T14:02:16.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.374+0000 7fe2f8a17740 1 Processor -- start 2024-04-25T14:02:16.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ee0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2eb4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2d88fe0 con 0x55d3d2ada590 2024-04-25T14:02:16.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2eb3d00 con 0x55d3d2ad23b0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55d3d2ee1860 con 0x55d3d2afb270 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2f8a17740 10 client.0 _ll_register_callbacks cb 0x55d3d2d9f150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36608/0 (socket says 172.21.15.150:36608) 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 -- 172.21.15.150:0/3218180402 learned_addr learned my addr 172.21.15.150:0/3218180402 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ee0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2eb4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 -- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 msgr2=0x55d3d2eb4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2eb4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 -- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 msgr2=0x55d3d2ee0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ee0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 -- 172.21.15.150:0/3218180402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3d2b132d0 con 0x55d3d2ada590 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ee0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0035040 tx=0x7fe2e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dffab0d806fbad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.375+0000 7fe2ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0041020 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2e003d670 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2dc026070 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fe2e004b050 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 10 client.4425 fetch_fsmap learned FSMap version 6 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55d3d2bdd6b0 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc026070 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fe2e004e020 con 0x55d3d2ada590 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 10 client.4425 fetch_fsmap finished waiting for FSMap version 6 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 10 client.4425 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 20 client.4425 populate_metadata read hostname 'smithi150' 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2ab9e40 0x55d3d2ef4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 4 client.4425 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:02:16.382 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 --> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3d2e0f350 con 0x55d3d2ab9e40 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.376+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2ab9e40 0x55d3d2ef4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.377+0000 7fe2ca7fc640 10 client.4425 ms_handle_connect on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.377+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2ab9e40 0x55d3d2ef4f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=665110ee8f1aa1a3 server_cookie=9553a3eaf71107a5 in_seq=0 out_seq=0 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.377+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/3218180402 <== mds.0 v2:172.21.15.187:6833/3860691734 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d3d2e0f350 con 0x55d3d2ab9e40 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.377+0000 7fe2ca7fc640 10 client.4425 handle_command_reply: tid=0 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 client.4425 shutdown 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 msgr2=0x55d3d2eb4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0035040 tx=0x7fe2e0004730 comp rx=0 tx=0).stop 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 shutdown_connections 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2eb4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2eb4610 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2ab9e40 0x55d3d2ef4f40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/3218180402 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ad23b0 0x55d3d2ee0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 >> 172.21.15.150:0/3218180402 conn(0x55d3d2aef3c0 msgr2=0x55d3d2e9c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2ca7fc640 0 client.4425 ms_handle_reset on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 shutdown_connections 2024-04-25T14:02:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 1 -- 172.21.15.150:0/3218180402 wait complete. 2024-04-25T14:02:16.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.378+0000 7fe2f8a17740 20 client.4425 trim_cache size 0 max 16384 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.396+0000 7fe2f8a17740 1 Processor -- start 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.396+0000 7fe2f8a17740 1 -- start start 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 0x55d3d2ec77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2efc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec203130 con 0x55d3d2ad23b0 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe2ec2024c0 con 0x55d3d2ada590 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe2ec202b10 con 0x55d3d2afb270 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36614/0 (socket says 172.21.15.150:36614) 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/1100650372 learned_addr learned my addr 172.21.15.150:0/1100650372 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 0x55d3d2ec77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2efc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 msgr2=0x55d3d2efc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2efc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 msgr2=0x55d3d2ec77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 0x55d3d2ec77f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 -- 172.21.15.150:0/1100650372 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3d2e98440 con 0x55d3d2ad23b0 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2efc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:16.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 0x55d3d2ec77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009960 tx=0x7fe2dc018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=111d4856ee7c031 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2d0ff9640 1 -- 172.21.15.150:0/1100650372 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc03d070 con 0x55d3d2ad23b0 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2d0ff9640 1 -- 172.21.15.150:0/1100650372 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc026040 con 0x55d3d2ad23b0 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.398+0000 7fe2d0ff9640 1 -- 172.21.15.150:0/1100650372 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2dc0218d0 con 0x55d3d2ad23b0 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 -- 172.21.15.150:0/1100650372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 msgr2=0x55d3d2eb32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc009960 tx=0x7fe2dc018000 comp rx=0 tx=0).stop 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 -- 172.21.15.150:0/1100650372 shutdown_connections 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2afb270 0x55d3d2efc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2ada590 0x55d3d2ec77f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/1100650372 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ad23b0 0x55d3d2eb32b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 -- 172.21.15.150:0/1100650372 >> 172.21.15.150:0/1100650372 conn(0x55d3d2aef3c0 msgr2=0x55d3d2f06130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 -- 172.21.15.150:0/1100650372 shutdown_connections 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.399+0000 7fe2f8a17740 1 -- 172.21.15.150:0/1100650372 wait complete. 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.400+0000 7fe2f8a17740 1 -- start start 2024-04-25T14:02:16.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.400+0000 7fe2f8a17740 1 Processor -- start 2024-04-25T14:02:16.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ad23b0 0x55d3d2ed84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2afb270 0x55d3d2eef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2ef0710 con 0x55d3d2ada590 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55d3d2ed98e0 con 0x55d3d2afb270 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55d3d2ef0d60 con 0x55d3d2ad23b0 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.401+0000 7fe2f8a17740 10 client.0 _ll_register_callbacks cb 0x7fe2e0035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36628/0 (socket says 172.21.15.150:36628) 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 -- 172.21.15.150:0/865629098 learned_addr learned my addr 172.21.15.150:0/865629098 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ad23b0 0x55d3d2ed84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ebfff640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2afb270 0x55d3d2eef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 -- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ad23b0 msgr2=0x55d3d2ed84a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ad23b0 0x55d3d2ed84a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 -- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2afb270 msgr2=0x55d3d2eef270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2afb270 0x55d3d2eef270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 -- 172.21.15.150:0/865629098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3d2eeffb0 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2eaffd640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097a0 tx=0x7fe2e0009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=384d179bb95e6425 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2e0054060 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe2e00463f0 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2dc026040 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fe2e005a050 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2f8a17740 10 client.4431 fetch_fsmap learned FSMap version 6 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55d3d2f630c0 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2dc026040 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fe2e005d020 con 0x55d3d2ada590 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.402+0000 7fe2f8a17740 10 client.4431 fetch_fsmap finished waiting for FSMap version 6 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 10 client.4431 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 20 client.4431 populate_metadata read hostname 'smithi150' 2024-04-25T14:02:16.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2e96300 0x55d3d2e96710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 4 client.4431 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 --> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] -- command(tid 0: {"prefix": "status"}) -- 0x55d3d2e091c0 con 0x55d3d2e96300 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2e96300 0x55d3d2e96710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2ca7fc640 10 client.4431 ms_handle_connect on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2eb7fe640 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2e96300 0x55d3d2e96710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f243f7c0cf77116 server_cookie=165cb55767988a62 in_seq=0 out_seq=0 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2ca7fc640 1 -- 172.21.15.150:0/865629098 <== mds.0 v2:172.21.15.187:6833/3860691734 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55d3d2e091c0 con 0x55d3d2e96300 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2ca7fc640 10 client.4431 handle_command_reply: tid=0 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 1 client.4431 shutdown 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 msgr2=0x55d3d2ed89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.403+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe2e00097a0 tx=0x7fe2e0009190 comp rx=0 tx=0).stop 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 shutdown_connections 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x55d3d2e96300 0x55d3d2e96710 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55d3d2afb270 0x55d3d2eef270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55d3d2ada590 0x55d3d2ed89e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 --2- 172.21.15.150:0/865629098 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55d3d2ad23b0 0x55d3d2ed84a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2ca7fc640 0 client.4431 ms_handle_reset on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 >> 172.21.15.150:0/865629098 conn(0x55d3d2aef3c0 msgr2=0x55d3d2f061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 shutdown_connections 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 1 -- 172.21.15.150:0/865629098 wait complete. 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f8a17740 20 client.4431 trim_cache size 0 max 16384 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:02:16.411 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4293, 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 7.2508040530000004, 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 6, 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 24, 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 23, 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 14.103460317 2024-04-25T14:02:16.412 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 msgr2=0x7fe2d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 0x7fe2d80769c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec06e7e0 tx=0x7fe2dc008040 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 msgr2=0x7fe2ec0cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe2e400c730 tx=0x7fe2e400cbf0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 shutdown_connections 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe2d80744b0 0x7fe2d80769c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe2ec0cd120 0x7fe2ec0cd530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe2ec163d80 0x7fe2ec14cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 --2- 172.21.15.150:0/601023568 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe2ec0b5900 0x7fe2ec0b78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 >> 172.21.15.150:0/601023568 conn(0x7fe2ec0d03c0 msgr2=0x7fe2ec0d16b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 shutdown_connections 2024-04-25T14:02:16.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.404+0000 7fe2f1b6c640 1 -- 172.21.15.150:0/601023568 wait complete. 2024-04-25T14:02:16.415 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T14:02:16.416 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:16.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:02:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.523+0000 7fd92b900640 1 Processor -- start 2024-04-25T14:02:16.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.523+0000 7fd92b900640 1 -- start start 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd92b900640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 0x7fd924160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd92b900640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9240b53a0 0x7fd9240b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd92b900640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9240b5cf0 con 0x7fd9240b2b10 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd92b900640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd9240b7db0 con 0x7fd9240b53a0 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd92b900640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd924164690 con 0x7fd9240d2e90 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.524+0000 7fd929e9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9240b53a0 0x7fd9240b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.525 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9240b53a0 0x7fd9240b57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36630/0 (socket says 172.21.15.150:36630) 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 -- 172.21.15.150:0/3218274861 learned_addr learned my addr 172.21.15.150:0/3218274861 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd928e9b640 1 --2- 172.21.15.150:0/3218274861 >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 0x7fd924160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 -- 172.21.15.150:0/3218274861 >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 msgr2=0x7fd924160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 --2- 172.21.15.150:0/3218274861 >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 0x7fd924160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 -- 172.21.15.150:0/3218274861 >> v1:172.21.15.187:6789/0 conn(0x7fd9240b2b10 legacy=0x7fd9240cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 -- 172.21.15.150:0/3218274861 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd924164bc0 con 0x7fd9240b53a0 2024-04-25T14:02:16.526 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd928e9b640 1 --2- 172.21.15.150:0/3218274861 >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 0x7fd924160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:16.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.525+0000 7fd929e9d640 1 --2- 172.21.15.150:0/3218274861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9240b53a0 0x7fd9240b57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009e50 tx=0x7fd91c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be45d61c3cc34a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd91bfff640 1 -- 172.21.15.150:0/3218274861 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c036070 con 0x7fd9240b53a0 2024-04-25T14:02:16.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd91bfff640 1 -- 172.21.15.150:0/3218274861 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91c031600 con 0x7fd9240b53a0 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd91bfff640 1 -- 172.21.15.150:0/3218274861 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0319d0 con 0x7fd9240b53a0 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 -- 172.21.15.150:0/3218274861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9240b53a0 msgr2=0x7fd9240b57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 --2- 172.21.15.150:0/3218274861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9240b53a0 0x7fd9240b57b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009e50 tx=0x7fd91c02a640 comp rx=0 tx=0).stop 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 -- 172.21.15.150:0/3218274861 shutdown_connections 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 --2- 172.21.15.150:0/3218274861 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9240b53a0 0x7fd9240b57b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 --2- 172.21.15.150:0/3218274861 >> v2:172.21.15.187:3300/0 conn(0x7fd9240d2e90 0x7fd924160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 -- 172.21.15.150:0/3218274861 >> 172.21.15.150:0/3218274861 conn(0x7fd9240d03a0 msgr2=0x7fd9240d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.526+0000 7fd92b900640 1 -- 172.21.15.150:0/3218274861 shutdown_connections 2024-04-25T14:02:16.528 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.527+0000 7fd92b900640 1 -- 172.21.15.150:0/3218274861 wait complete. 2024-04-25T14:02:16.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.528+0000 7fd92b900640 1 Processor -- start 2024-04-25T14:02:16.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.528+0000 7fd92b900640 1 -- start start 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9240b53a0 0x7fd924158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9240d2e90 0x7fd924149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9241648e0 con 0x7fd92406fb30 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9240b8350 con 0x7fd9240b53a0 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.529+0000 7fd92b900640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9240b7fe0 con 0x7fd9240d2e90 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:36646/0 (socket says 172.21.15.150:36646) 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 -- 172.21.15.150:0/426559484 learned_addr learned my addr 172.21.15.150:0/426559484 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd92969c640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9240b53a0 0x7fd924158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd928e9b640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9240d2e90 0x7fd924149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 -- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9240d2e90 msgr2=0x7fd924149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9240d2e90 0x7fd924149e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 -- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9240b53a0 msgr2=0x7fd924158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9240b53a0 0x7fd924158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 -- 172.21.15.150:0/426559484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd924070480 con 0x7fd92406fb30 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd929e9d640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009e50 tx=0x7fd91c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c036050 con 0x7fd92406fb30 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd91c042070 con 0x7fd92406fb30 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9241589a0 con 0x7fd92406fb30 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.530+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd91c009510 con 0x7fd92406fb30 2024-04-25T14:02:16.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.531+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd92415b650 con 0x7fd92406fb30 2024-04-25T14:02:16.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.531+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9240b51c0 con 0x7fd92406fb30 2024-04-25T14:02:16.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.532+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd91c007bd0 con 0x7fd92406fb30 2024-04-25T14:02:16.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.533+0000 7fd919ffb640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 0x7fd9080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:16.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.533+0000 7fd92969c640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 0x7fd9080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:16.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.533+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd91c0b62e0 con 0x7fd92406fb30 2024-04-25T14:02:16.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.533+0000 7fd92969c640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 0x7fd9080769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9200016e0 tx=0x7fd920009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:16.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.537+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9240b51c0 con 0x7fd92406fb30 2024-04-25T14:02:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.751+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd924076a40 con 0x7fd92406fb30 2024-04-25T14:02:16.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.752+0000 7fd919ffb640 1 -- 172.21.15.150:0/426559484 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7fd924070480 con 0x7fd92406fb30 2024-04-25T14:02:16.752 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T14:02:16.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 msgr2=0x7fd9080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 0x7fd9080769c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd9200016e0 tx=0x7fd920009040 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 msgr2=0x7fd92406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009e50 tx=0x7fd91c004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 shutdown_connections 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9080744b0 0x7fd9080769c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd92406fb30 0x7fd92406ff40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9240d2e90 0x7fd924149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 --2- 172.21.15.150:0/426559484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9240b53a0 0x7fd924158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 >> 172.21.15.150:0/426559484 conn(0x7fd9240d03a0 msgr2=0x7fd92415f530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 shutdown_connections 2024-04-25T14:02:16.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16.754+0000 7fd92b900640 1 -- 172.21.15.150:0/426559484 wait complete. 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:02:16.765 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T14:02:16.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:16.766 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-04-25T14:02:16.780 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:16.780 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-04-25T14:02:16.781 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42763sec preferred_lft 42763sec 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::eb5b:7bab:e07b:2aac/64 scope link noprefixroute 2024-04-25T14:02:16.782 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:16.783 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T14:02:16.783 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:16.783 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:16.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-04-25T14:02:16.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-04-25T14:02:16.783 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-04-25T14:02:16.784 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T14:02:16.784 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:16.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:16.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:16.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:16.957 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:02:16.983 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-25T14:02:16.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:16.984 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-04-25T14:02:17.055 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-04-25T14:02:17.056 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:02:17.056 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:02:17.056 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:02:17.056 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:02:17.057 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:17.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:17.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:17.335 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:02:17.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:17.355 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-04-25T14:02:17.427 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:17.428 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:17.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:17.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-25T14:02:17.428 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:17.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:17.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:17.548 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-25T14:02:17.549 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:17.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:17.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:17.671 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:17.671 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:17.671 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-04-25T14:02:17.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T14:02:17.672 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:17.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:17.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:17.761 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-25T14:02:17.761 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:17.761 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:17.797 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-25T14:02:17.797 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:17.797 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:17.861 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-04-25T14:02:17.932 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-04-25T14:02:17.993 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-04-25T14:02:17.994 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T14:02:17.995 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:17.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:17.996 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-04-25T14:02:18.050 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-25T14:02:18.051 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-25T14:02:18.097 DEBUG:teuthology.orchestra.run.smithi150:> sudo modprobe fuse 2024-04-25T14:02:18.122 DEBUG:teuthology.orchestra.run.smithi150:> cat /proc/self/mounts | awk '{print $2}' 2024-04-25T14:02:18.179 INFO:teuthology.orchestra.run.smithi150.stdout:/proc 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/dev 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/security 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/shm 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/pts 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/run 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/cgroup 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/pstore 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/bpf 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/ 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/selinux 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/proc/sys/fs/binfmt_misc 2024-04-25T14:02:18.180 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/hugepages 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/dev/mqueue 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/debug 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/tracing 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysusers.service 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/kernel/config 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/sys/fs/fuse/connections 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-sysctl.service 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/nfs/rpc_pipefs 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/run/user/1000 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph 2024-04-25T14:02:18.181 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-0 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-1 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-2 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/var/lib/ceph/osd/ceph-3 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run.smithi150.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:18.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:18.182 DEBUG:teuthology.orchestra.run.smithi150:> ls /sys/fs/fuse/connections 2024-04-25T14:02:18.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:2024-04-25T14:02:18.198+0000 7fdff260b480 -1 init, newargv = 0x556e493f16a0 newargc=15 2024-04-25T14:02:18.198 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[42164]: starting ceph client 2024-04-25T14:02:18.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[42164]: starting fuse 2024-04-25T14:02:18.217 INFO:teuthology.orchestra.run.smithi150.stdout:43 2024-04-25T14:02:18.217 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> sudo stdin-killer -- python3 -c ' 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> import glob 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> import re 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> import subprocess 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> def _find_admin_socket(client_name): 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> files = glob.glob(asok_path) 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:02:18.218 DEBUG:teuthology.orchestra.run.smithi150:> # Given a non-glob path, it better be there 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> if "*" not in asok_path: 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> assert(len(files) == 1) 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> return files[0] 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> for f in files: 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> contents = proc_f.read() 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> if mountpoint in contents: 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> return f 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> print(_find_admin_socket("client.admin")) 2024-04-25T14:02:18.219 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:02:18.337 INFO:teuthology.orchestra.run.smithi150.stdout:/var/run/ceph/ceph-client.admin.42164.asok 2024-04-25T14:02:18.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:18.347 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.42164.asok 2024-04-25T14:02:18.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:18.348 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.42164.asok status 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "metadata": { 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_sha1": "994e2f20102a20fa9a27966c14b625cc01884df8", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "entity_id": "admin", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "hostname": "smithi150", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "pid": "42164", 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: "root": "/" 2024-04-25T14:02:18.489 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_count": 0, 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "dentry_pinned_count": 0, 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4446, 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "inst": { 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "name": { 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "client", 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "num": 4446 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3030184617 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:02:18.490 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": { 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "type": "v1", 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "addr": "192.168.0.1:0", 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "nonce": 3030184617 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "inst_str": "client.4446 192.168.0.1:0/3030184617", 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "addr_str": "192.168.0.1:0/3030184617", 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "inode_count": 1, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "mds_epoch": 6, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch": 24, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "osd_epoch_barrier": 0, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "blocklisted": false, 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs" 2024-04-25T14:02:18.491 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:02:18.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:18.497 DEBUG:teuthology.orchestra.run.smithi150:> stat --file-system '--printf=%T 2024-04-25T14:02:18.497 DEBUG:teuthology.orchestra.run.smithi150:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:18.514 INFO:teuthology.orchestra.run.smithi150.stdout:fuseblk 2024-04-25T14:02:18.514 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:18.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:18.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:18.581 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-25T14:02:18.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:18.691 INFO:teuthology.orchestra.run.smithi150.stdout:# file: . 2024-04-25T14:02:18.691 INFO:teuthology.orchestra.run.smithi150.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-25T14:02:18.691 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:18.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:18.698 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:18.698 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:18.714 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:18.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.861+0000 7fd085caf640 1 Processor -- start 2024-04-25T14:02:18.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.861+0000 7fd085caf640 1 -- start start 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd085caf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0800d1120 0x7fd080070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd085caf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd080154950 0x7fd080154d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd085caf640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0800aec30 con 0x7fd080161b10 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd085caf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd0800d1530 con 0x7fd080154950 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd085caf640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0800aef10 con 0x7fd0800d1120 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd080154950 0x7fd080154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd080154950 0x7fd080154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47268/0 (socket says 172.21.15.150:47268) 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 -- 172.21.15.150:0/1475319613 learned_addr learned my addr 172.21.15.150:0/1475319613 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07f7fe640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0800d1120 0x7fd080070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 -- 172.21.15.150:0/1475319613 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0800d1120 msgr2=0x7fd080070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0800d1120 0x7fd080070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 -- 172.21.15.150:0/1475319613 >> v1:172.21.15.187:6789/0 conn(0x7fd080161b10 legacy=0x7fd080161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:18.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.863+0000 7fd07ffff640 1 -- 172.21.15.150:0/1475319613 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0800af1f0 con 0x7fd080154950 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.864+0000 7fd07ffff640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080154950 0x7fd080154d60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd078009e50 tx=0x7fd07802a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392a32051dcc0407 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.864+0000 7fd07e7fc640 1 -- 172.21.15.150:0/1475319613 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078036070 con 0x7fd080154950 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.864+0000 7fd07e7fc640 1 -- 172.21.15.150:0/1475319613 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd078031600 con 0x7fd080154950 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.864+0000 7fd07e7fc640 1 -- 172.21.15.150:0/1475319613 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0780319d0 con 0x7fd080154950 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 -- 172.21.15.150:0/1475319613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080154950 msgr2=0x7fd080154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080154950 0x7fd080154d60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd078009e50 tx=0x7fd07802a5b0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 -- 172.21.15.150:0/1475319613 shutdown_connections 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080154950 0x7fd080154d60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 --2- 172.21.15.150:0/1475319613 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0800d1120 0x7fd080070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 -- 172.21.15.150:0/1475319613 >> 172.21.15.150:0/1475319613 conn(0x7fd0800d1ae0 msgr2=0x7fd0800d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:18.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 -- 172.21.15.150:0/1475319613 shutdown_connections 2024-04-25T14:02:18.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.865+0000 7fd085caf640 1 -- 172.21.15.150:0/1475319613 wait complete. 2024-04-25T14:02:18.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.867+0000 7fd085caf640 1 Processor -- start 2024-04-25T14:02:18.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.867+0000 7fd085caf640 1 -- start start 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800d1120 0x7fd0801f8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd080154950 0x7fd0801f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0801552a0 con 0x7fd080161b10 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd080071430 con 0x7fd0800d1120 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd085caf640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd08015b4f0 con 0x7fd080154950 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd07ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd07ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47278/0 (socket says 172.21.15.150:47278) 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd07ffff640 1 -- 172.21.15.150:0/971663641 learned_addr learned my addr 172.21.15.150:0/971663641 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd07f7fe640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800d1120 0x7fd0801f8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.868+0000 7fd07effd640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd080154950 0x7fd0801f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 -- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd080154950 msgr2=0x7fd0801f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd080154950 0x7fd0801f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 -- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800d1120 msgr2=0x7fd0801f8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800d1120 0x7fd0801f8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 -- 172.21.15.150:0/971663641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0801f59e0 con 0x7fd080161b10 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd07ffff640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd07802aba0 tx=0x7fd078004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd07803f020 con 0x7fd080161b10 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd078037040 con 0x7fd080161b10 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0801f4d80 con 0x7fd080161b10 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd078036070 con 0x7fd080161b10 2024-04-25T14:02:18.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.869+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd080146e00 con 0x7fd080161b10 2024-04-25T14:02:18.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.870+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd080076a40 con 0x7fd080161b10 2024-04-25T14:02:18.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.871+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd078042020 con 0x7fd080161b10 2024-04-25T14:02:18.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.871+0000 7fd067fff640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 0x7fd06c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:18.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.871+0000 7fd07f7fe640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 0x7fd06c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:18.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.871+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd0780b5910 con 0x7fd080161b10 2024-04-25T14:02:18.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.872+0000 7fd07f7fe640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 0x7fd06c0769f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd070006fd0 tx=0x7fd070008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:18.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:18.876+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd078049070 con 0x7fd080161b10 2024-04-25T14:02:19.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.054+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd080070250 con 0x7fd080161b10 2024-04-25T14:02:19.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.055+0000 7fd067fff640 1 -- 172.21.15.150:0/971663641 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fd07807fd60 con 0x7fd080161b10 2024-04-25T14:02:19.056 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:19.056 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:19.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 msgr2=0x7fd06c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 0x7fd06c0769f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd070006fd0 tx=0x7fd070008040 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 msgr2=0x7fd080145fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd07802aba0 tx=0x7fd078004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 shutdown_connections 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd06c0744e0 0x7fd06c0769f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080161b10 0x7fd080145fb0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd080154950 0x7fd0801f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 --2- 172.21.15.150:0/971663641 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800d1120 0x7fd0801f8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 >> 172.21.15.150:0/971663641 conn(0x7fd0800d1ae0 msgr2=0x7fd0800b1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 shutdown_connections 2024-04-25T14:02:19.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.057+0000 7fd085caf640 1 -- 172.21.15.150:0/971663641 wait complete. 2024-04-25T14:02:19.069 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:19.070 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:19.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.179+0000 7f513e1c7640 1 Processor -- start 2024-04-25T14:02:19.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.179+0000 7f513e1c7640 1 -- start start 2024-04-25T14:02:19.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f513e1c7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f51381603d0 0x7f51380b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f513e1c7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51381607e0 con 0x7f51380735b0 2024-04-25T14:02:19.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f513e1c7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5138164a50 con 0x7f51380b6800 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f513e1c7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f51380d2e90 con 0x7f51381603d0 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5136ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f51380735b0 0x7f51380b8300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49948/0 (socket says 172.21.15.150:49948) 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5136ffd640 1 -- 172.21.15.150:0/344872403 learned_addr learned my addr 172.21.15.150:0/344872403 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5137fff640 1 --2- 172.21.15.150:0/344872403 >> v2:172.21.15.150:3300/0 conn(0x7f51381603d0 0x7f51380b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f51367fc640 1 -- 172.21.15.150:0/344872403 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3871556179 0 0) 0x7f51381607e0 con 0x7f51380735b0 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f51367fc640 1 -- 172.21.15.150:0/344872403 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5120003520 con 0x7f51380735b0 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5137fff640 1 -- 172.21.15.150:0/344872403 >> v1:172.21.15.187:6789/0 conn(0x7f51380b6800 legacy=0x7f5138072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5137fff640 1 -- 172.21.15.150:0/344872403 >> v1:172.21.15.150:6789/0 conn(0x7f51380735b0 legacy=0x7f51380b8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.181+0000 7f5137fff640 1 -- 172.21.15.150:0/344872403 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5138164d30 con 0x7f51381603d0 2024-04-25T14:02:19.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.182+0000 7f5137fff640 1 --2- 172.21.15.150:0/344872403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f51380b8a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5130009d90 tx=0x7f513002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a821fbcffdba2f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.182+0000 7f51367fc640 1 -- 172.21.15.150:0/344872403 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130007740 con 0x7f51381603d0 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.182+0000 7f51367fc640 1 -- 172.21.15.150:0/344872403 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51300078e0 con 0x7f51381603d0 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.182+0000 7f51367fc640 1 -- 172.21.15.150:0/344872403 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130007c90 con 0x7f51381603d0 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 -- 172.21.15.150:0/344872403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 msgr2=0x7f51380b8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 --2- 172.21.15.150:0/344872403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f51380b8a10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5130009d90 tx=0x7f513002a660 comp rx=0 tx=0).stop 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 -- 172.21.15.150:0/344872403 shutdown_connections 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 --2- 172.21.15.150:0/344872403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f51380b8a10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 -- 172.21.15.150:0/344872403 >> 172.21.15.150:0/344872403 conn(0x7f51380d39b0 msgr2=0x7f51380d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 -- 172.21.15.150:0/344872403 shutdown_connections 2024-04-25T14:02:19.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.183+0000 7f513e1c7640 1 -- 172.21.15.150:0/344872403 wait complete. 2024-04-25T14:02:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.185+0000 7f513e1c7640 1 Processor -- start 2024-04-25T14:02:19.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.185+0000 7f513e1c7640 1 -- start start 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51380b6800 0x7f5138070440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5138151f00 0x7f5138152310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51380cfd10 con 0x7f51381603d0 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f51380d0080 con 0x7f5138151f00 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f513e1c7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f51380b98e0 con 0x7f51380b6800 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f5136ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f5136ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47302/0 (socket says 172.21.15.150:47302) 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f5136ffd640 1 -- 172.21.15.150:0/825798616 learned_addr learned my addr 172.21.15.150:0/825798616 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f5137fff640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5138151f00 0x7f5138152310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.186+0000 7f51377fe640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51380b6800 0x7f5138070440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 -- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51380b6800 msgr2=0x7f5138070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51380b6800 0x7f5138070440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 -- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5138151f00 msgr2=0x7f5138152310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5138151f00 0x7f5138152310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 -- 172.21.15.150:0/825798616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5138152850 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f5136ffd640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c3f0 tx=0x7f512c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c017070 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f512c005d00 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5138160c20 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512c007310 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.187+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51381611e0 con 0x7f51381603d0 2024-04-25T14:02:19.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.188+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5138161900 con 0x7f51381603d0 2024-04-25T14:02:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.189+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f512c01a070 con 0x7f51381603d0 2024-04-25T14:02:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.189+0000 7f511ffff640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 0x7f51240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.189+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f512c0947d0 con 0x7f51381603d0 2024-04-25T14:02:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.189+0000 7f51377fe640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 0x7f51240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.189+0000 7f51377fe640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 0x7f51240769c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51380d3910 tx=0x7f5128009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.194+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5138152850 con 0x7f51381603d0 2024-04-25T14:02:19.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.367+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f51381609b0 con 0x7f51381603d0 2024-04-25T14:02:19.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.368+0000 7f511ffff640 1 -- 172.21.15.150:0/825798616 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f512c05e7b0 con 0x7f51381603d0 2024-04-25T14:02:19.368 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:19.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 msgr2=0x7f51240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 0x7f51240769c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f51380d3910 tx=0x7f5128009040 comp rx=0 tx=0).stop 2024-04-25T14:02:19.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 msgr2=0x7f513814a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f512c00c3f0 tx=0x7f512c00c8b0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 shutdown_connections 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f51240744b0 0x7f51240769c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5138151f00 0x7f5138152310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f51381603d0 0x7f513814a140 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 --2- 172.21.15.150:0/825798616 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51380b6800 0x7f5138070440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 >> 172.21.15.150:0/825798616 conn(0x7f51380d39b0 msgr2=0x7f51380b4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 shutdown_connections 2024-04-25T14:02:19.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.370+0000 7f513e1c7640 1 -- 172.21.15.150:0/825798616 wait complete. 2024-04-25T14:02:19.381 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.491+0000 7ffb4d225640 1 Processor -- start 2024-04-25T14:02:19.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.491+0000 7ffb4d225640 1 -- start start 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4d225640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffb481624a0 0x7ffb480d2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4d225640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4815b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4d225640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb480d1e00 con 0x7ffb480d2a40 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4d225640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ffb480d3390 con 0x7ffb481624a0 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4d225640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb4815fa00 con 0x7ffb4815b4c0 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4659c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffb481624a0 0x7ffb480d2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb4759e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4815b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb46d9d640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ffb480d2a40 0x7ffb48161d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:60488/0 (socket says 172.21.15.150:60488) 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.492+0000 7ffb46d9d640 1 -- 172.21.15.150:0/916471693 learned_addr learned my addr 172.21.15.150:0/916471693 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4659c640 1 -- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 msgr2=0x7ffb4815b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:19.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4659c640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4815b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4659c640 1 -- 172.21.15.150:0/916471693 >> v1:172.21.15.187:6789/0 conn(0x7ffb480d2a40 legacy=0x7ffb48161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4659c640 1 -- 172.21.15.150:0/916471693 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb4815fce0 con 0x7ffb481624a0 2024-04-25T14:02:19.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb45d9b640 1 -- 172.21.15.150:0/916471693 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 166549295 0 0) 0x7ffb480d1e00 con 0x7ffb480d2a40 2024-04-25T14:02:19.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4759e640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.493+0000 7ffb4659c640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb481624a0 0x7ffb480d2e50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0097a0 tx=0x7ffb3c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1e53ed15b6c72ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb45d9b640 1 -- 172.21.15.150:0/916471693 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c037070 con 0x7ffb481624a0 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb45d9b640 1 -- 172.21.15.150:0/916471693 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb3c032440 con 0x7ffb481624a0 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb45d9b640 1 -- 172.21.15.150:0/916471693 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c032810 con 0x7ffb481624a0 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 -- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb481624a0 msgr2=0x7ffb480d2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb481624a0 0x7ffb480d2e50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c0097a0 tx=0x7ffb3c02a180 comp rx=0 tx=0).stop 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 -- 172.21.15.150:0/916471693 shutdown_connections 2024-04-25T14:02:19.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4815b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 --2- 172.21.15.150:0/916471693 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb481624a0 0x7ffb480d2e50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 -- 172.21.15.150:0/916471693 >> 172.21.15.150:0/916471693 conn(0x7ffb4809b4a0 msgr2=0x7ffb4809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 -- 172.21.15.150:0/916471693 shutdown_connections 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.494+0000 7ffb4d225640 1 -- 172.21.15.150:0/916471693 wait complete. 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.496+0000 7ffb4d225640 1 Processor -- start 2024-04-25T14:02:19.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.496+0000 7ffb4d225640 1 -- start start 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4814edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb4809fb00 0x7ffb4809ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb480d35f0 con 0x7ffb480d2a40 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffb481602a0 con 0x7ffb4809fb00 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4d225640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb4815be10 con 0x7ffb4815b4c0 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47304/0 (socket says 172.21.15.150:47304) 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 -- 172.21.15.150:0/4240284037 learned_addr learned my addr 172.21.15.150:0/4240284037 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4659c640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4814edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb4759e640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb4809fb00 0x7ffb4809ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 -- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 msgr2=0x7ffb4814edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4814edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 -- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb4809fb00 msgr2=0x7ffb4809ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb4809fb00 0x7ffb4809ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 -- 172.21.15.150:0/4240284037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb480a0450 con 0x7ffb480d2a40 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb46d9d640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c440 tx=0x7ffb3800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.498+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38018070 con 0x7ffb480d2a40 2024-04-25T14:02:19.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.499+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb380074e0 con 0x7ffb480d2a40 2024-04-25T14:02:19.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.499+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb4814c4d0 con 0x7ffb480d2a40 2024-04-25T14:02:19.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.499+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb38004a20 con 0x7ffb480d2a40 2024-04-25T14:02:19.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.499+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb4809c820 con 0x7ffb480d2a40 2024-04-25T14:02:19.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.499+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb4814c6a0 con 0x7ffb480d2a40 2024-04-25T14:02:19.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.500+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffb3801b070 con 0x7ffb480d2a40 2024-04-25T14:02:19.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.501+0000 7ffb2f7fe640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 0x7ffb340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.501+0000 7ffb4659c640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 0x7ffb340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.501+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffb38094790 con 0x7ffb480d2a40 2024-04-25T14:02:19.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.501+0000 7ffb4659c640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 0x7ffb340769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c034000 tx=0x7ffb3c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.505+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb480a0450 con 0x7ffb480d2a40 2024-04-25T14:02:19.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.681+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ffb48076a40 con 0x7ffb480d2a40 2024-04-25T14:02:19.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.681+0000 7ffb2f7fe640 1 -- 172.21.15.150:0/4240284037 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7ffb3805ebe0 con 0x7ffb480d2a40 2024-04-25T14:02:19.682 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:19.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 msgr2=0x7ffb340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 0x7ffb340769f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c034000 tx=0x7ffb3c02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 msgr2=0x7ffb4815a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c440 tx=0x7ffb3800c900 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 shutdown_connections 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb340744e0 0x7ffb340769f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffb4809fb00 0x7ffb4809ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb4815b4c0 0x7ffb4814edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 --2- 172.21.15.150:0/4240284037 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb480d2a40 0x7ffb4815a0f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 >> 172.21.15.150:0/4240284037 conn(0x7ffb4809b4a0 msgr2=0x7ffb4806d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 shutdown_connections 2024-04-25T14:02:19.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.683+0000 7ffb4d225640 1 -- 172.21.15.150:0/4240284037 wait complete. 2024-04-25T14:02:19.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:19.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.805+0000 7f3a69451640 1 Processor -- start 2024-04-25T14:02:19.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.805+0000 7f3a69451640 1 -- start start 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.806+0000 7f3a69451640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a6416b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.806+0000 7f3a69451640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a640b9600 con 0x7f3a64160150 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.806+0000 7f3a69451640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a640d4f20 con 0x7f3a640b4c10 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a69451640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a6406df10 con 0x7f3a640b91f0 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a627fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3a64160150 0x7f3a6406d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49954/0 (socket says 172.21.15.150:49954) 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a627fc640 1 -- 172.21.15.150:0/1416378237 learned_addr learned my addr 172.21.15.150:0/1416378237 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a6416b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546953192 0 0) 0x7f3a640b9600 con 0x7f3a64160150 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a4c003520 con 0x7f3a64160150 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2193808294 0 0) 0x7f3a4c003520 con 0x7f3a64160150 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 msgr2=0x7f3a6416b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a6416b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 >> v1:172.21.15.187:6789/0 conn(0x7f3a640b4c10 legacy=0x7f3a6406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a640d5200 con 0x7f3a64160150 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f3a58001690 con 0x7f3a64160150 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 >> v1:172.21.15.150:6789/0 conn(0x7f3a64160150 legacy=0x7f3a6406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a4c0039e0 0x7f3a4c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a4c004800 0x7f3a4c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a4c005170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6406df10 con 0x7f3a4c004800 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4c00baa0 con 0x7f3a4c0039e0 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a4c00bca0 con 0x7f3a640b91f0 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.807+0000 7f3a627fc640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a4c004800 0x7f3a4c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a62ffd640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a4c0039e0 0x7f3a4c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a4c005170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 -- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 msgr2=0x7f3a4c005170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:19.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a4c005170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 -- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a4c0039e0 msgr2=0x7f3a4c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a4c0039e0 0x7f3a4c003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 -- 172.21.15.150:0/1416378237 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4c003520 con 0x7f3a4c004800 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a640b91f0 0x7f3a4c005170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.808+0000 7f3a627fc640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a4c004800 0x7f3a4c004c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3a640b97d0 tx=0x7f3a4c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db0ffa917fda53c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a58007d50 con 0x7f3a4c004800 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a58005ad0 con 0x7f3a4c004800 2024-04-25T14:02:19.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a61ffb640 1 -- 172.21.15.150:0/1416378237 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a58007070 con 0x7f3a4c004800 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 -- 172.21.15.150:0/1416378237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a4c004800 msgr2=0x7f3a4c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 --2- 172.21.15.150:0/1416378237 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a4c004800 0x7f3a4c004c10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3a640b97d0 tx=0x7f3a4c005830 comp rx=0 tx=0).stop 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a637fe640 1 -- 172.21.15.150:0/1416378237 reap_dead start 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 -- 172.21.15.150:0/1416378237 shutdown_connections 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 -- 172.21.15.150:0/1416378237 >> 172.21.15.150:0/1416378237 conn(0x7f3a640b7690 msgr2=0x7f3a640b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 -- 172.21.15.150:0/1416378237 shutdown_connections 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.809+0000 7f3a69451640 1 -- 172.21.15.150:0/1416378237 wait complete. 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.811+0000 7f3a69451640 1 Processor -- start 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.811+0000 7f3a69451640 1 -- start start 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 0x7f3a640ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a641fa980 0x7f3a640cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3a64160560 con 0x7f3a640b15e0 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6406cc60 con 0x7f3a641fa980 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.812+0000 7f3a69451640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3a6406c610 con 0x7f3a4c004800 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47334/0 (socket says 172.21.15.150:47334) 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 -- 172.21.15.150:0/1255818811 learned_addr learned my addr 172.21.15.150:0/1255818811 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a627fc640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a641fa980 0x7f3a640cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a62ffd640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 0x7f3a640ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 -- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 msgr2=0x7f3a640ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 0x7f3a640ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 -- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a641fa980 msgr2=0x7f3a640cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a641fa980 0x7f3a640cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 -- 172.21.15.150:0/1255818811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a640ace90 con 0x7f3a640b15e0 2024-04-25T14:02:19.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a62ffd640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 0x7f3a640ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a637fe640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c004780 tx=0x7f3a5c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c010e50 con 0x7f3a640b15e0 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a5c003990 con 0x7f3a640b15e0 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a640ad170 con 0x7f3a640b15e0 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c003d80 con 0x7f3a640b15e0 2024-04-25T14:02:19.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.813+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a640ad730 con 0x7f3a640b15e0 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.814+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a640acbe0 con 0x7f3a640b15e0 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.815+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3a5c01e020 con 0x7f3a640b15e0 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.815+0000 7f3a4b7fe640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 0x7f3a500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.815+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3a5c095d10 con 0x7f3a640b15e0 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.815+0000 7f3a62ffd640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 0x7f3a500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.816+0000 7f3a62ffd640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 0x7f3a500769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a540035f0 tx=0x7f3a54009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:19.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.820+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a5c060160 con 0x7f3a640b15e0 2024-04-25T14:02:19.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.994+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3a64076a40 con 0x7f3a640b15e0 2024-04-25T14:02:19.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.994+0000 7f3a4b7fe640 1 -- 172.21.15.150:0/1255818811 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f3a5c05fa60 con 0x7f3a640b15e0 2024-04-25T14:02:19.995 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 msgr2=0x7f3a500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 0x7f3a500769c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a540035f0 tx=0x7f3a54009040 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 msgr2=0x7f3a640b19f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c004780 tx=0x7f3a5c009f90 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 shutdown_connections 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3a500744b0 0x7f3a500769c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3a640b15e0 0x7f3a640b19f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.996+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3a641fa980 0x7f3a640cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.997+0000 7f3a69451640 1 --2- 172.21.15.150:0/1255818811 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3a4c004800 0x7f3a640ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.997+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 >> 172.21.15.150:0/1255818811 conn(0x7f3a640b7690 msgr2=0x7f3a640b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:19.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.997+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 shutdown_connections 2024-04-25T14:02:19.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:19.997+0000 7f3a69451640 1 -- 172.21.15.150:0/1255818811 wait complete. 2024-04-25T14:02:20.008 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:20.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.120+0000 7f1fe5001640 1 Processor -- start 2024-04-25T14:02:20.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.120+0000 7f1fe5001640 1 -- start start 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fe5001640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fe00cfb00 0x7f1fe00d2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fe5001640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 0x7f1fe00b3f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fe5001640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00d00a0 con 0x7f1fe00d2660 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fe5001640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe00ccd20 con 0x7f1fe00cfb00 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fe5001640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1fe00b1b40 con 0x7f1fe00b3b30 2024-04-25T14:02:20.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fded9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1fe00cfb00 0x7f1fe00d2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fd659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1fe00d2660 0x7f1fe00d2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49964/0 (socket says 172.21.15.150:49964) 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fd659c640 1 -- 172.21.15.150:0/1983817807 learned_addr learned my addr 172.21.15.150:0/1983817807 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.121+0000 7f1fdf59e640 1 --2- 172.21.15.150:0/1983817807 >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 0x7f1fe00b3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fded9d640 1 -- 172.21.15.150:0/1983817807 >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 msgr2=0x7f1fe00b3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fded9d640 1 --2- 172.21.15.150:0/1983817807 >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 0x7f1fe00b3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fded9d640 1 -- 172.21.15.150:0/1983817807 >> v1:172.21.15.150:6789/0 conn(0x7f1fe00d2660 legacy=0x7f1fe00d2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fde59c640 1 -- 172.21.15.150:0/1983817807 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157715298 0 0) 0x7f1fc8001110 con 0x7f1fe00d2660 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fded9d640 1 -- 172.21.15.150:0/1983817807 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe00ce9e0 con 0x7f1fe00cfb00 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fdf59e640 1 --2- 172.21.15.150:0/1983817807 >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 0x7f1fe00b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fded9d640 1 --2- 172.21.15.150:0/1983817807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00cfb00 0x7f1fe00d2040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00b8730 tx=0x7f1fd002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e4f7aebcf1427f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fde59c640 1 -- 172.21.15.150:0/1983817807 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0037070 con 0x7f1fe00cfb00 2024-04-25T14:02:20.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fde59c640 1 -- 172.21.15.150:0/1983817807 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd0032440 con 0x7f1fe00cfb00 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.122+0000 7f1fde59c640 1 -- 172.21.15.150:0/1983817807 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd002ae60 con 0x7f1fe00cfb00 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 -- 172.21.15.150:0/1983817807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00cfb00 msgr2=0x7f1fe00d2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 --2- 172.21.15.150:0/1983817807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00cfb00 0x7f1fe00d2040 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00b8730 tx=0x7f1fd002a110 comp rx=0 tx=0).stop 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 -- 172.21.15.150:0/1983817807 shutdown_connections 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 --2- 172.21.15.150:0/1983817807 >> v2:172.21.15.187:3300/0 conn(0x7f1fe00b3b30 0x7f1fe00b3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 --2- 172.21.15.150:0/1983817807 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00cfb00 0x7f1fe00d2040 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 -- 172.21.15.150:0/1983817807 >> 172.21.15.150:0/1983817807 conn(0x7f1fe009b4a0 msgr2=0x7f1fe009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 -- 172.21.15.150:0/1983817807 shutdown_connections 2024-04-25T14:02:20.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.123+0000 7f1fe5001640 1 -- 172.21.15.150:0/1983817807 wait complete. 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.125+0000 7f1fe5001640 1 Processor -- start 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.125+0000 7f1fe5001640 1 -- start start 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe00b3b30 0x7f1fe015e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe00cfb00 0x7f1fe0162de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00b3050 con 0x7f1fe00d2660 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1fe00cefa0 con 0x7f1fe00b3b30 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.126+0000 7f1fe5001640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1fe00b31d0 con 0x7f1fe00cfb00 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47360/0 (socket says 172.21.15.150:47360) 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 -- 172.21.15.150:0/3556243088 learned_addr learned my addr 172.21.15.150:0/3556243088 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fded9d640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe00b3b30 0x7f1fe015e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fd659c640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe00cfb00 0x7f1fe0162de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 -- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe00cfb00 msgr2=0x7f1fe0162de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe00cfb00 0x7f1fe0162de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 -- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe00b3b30 msgr2=0x7f1fe015e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe00b3b30 0x7f1fe015e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 -- 172.21.15.150:0/3556243088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe01fa900 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fdf59e640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cdb0 tx=0x7f1fd800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800f680 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fd800f840 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe01e8960 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800fc30 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.127+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fe01eadb0 con 0x7f1fe00d2660 2024-04-25T14:02:20.130 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.129+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fe0050740 con 0x7f1fe00d2660 2024-04-25T14:02:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.129+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1fd80213f0 con 0x7f1fe00d2660 2024-04-25T14:02:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.130+0000 7f1fd7fff640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 0x7f1fc00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.130+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1fd8096070 con 0x7f1fe00d2660 2024-04-25T14:02:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.130+0000 7f1fded9d640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 0x7f1fc00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.130+0000 7f1fded9d640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 0x7f1fc00769f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.134+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1fd8014030 con 0x7f1fe00d2660 2024-04-25T14:02:20.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.310+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1fe0068a70 con 0x7f1fe00d2660 2024-04-25T14:02:20.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.310+0000 7f1fd7fff640 1 -- 172.21.15.150:0/3556243088 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f1fd805ff00 con 0x7f1fe00d2660 2024-04-25T14:02:20.311 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:20.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 msgr2=0x7f1fc00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 0x7f1fc00769f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0034000 tx=0x7f1fd002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 msgr2=0x7f1fe01f9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cdb0 tx=0x7f1fd800a650 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 shutdown_connections 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1fc00744e0 0x7f1fc00769f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1fe00d2660 0x7f1fe01f9ab0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1fe00cfb00 0x7f1fe0162de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 --2- 172.21.15.150:0/3556243088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1fe00b3b30 0x7f1fe015e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 >> 172.21.15.150:0/3556243088 conn(0x7f1fe009b4a0 msgr2=0x7f1fe0071920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 shutdown_connections 2024-04-25T14:02:20.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.312+0000 7f1fe5001640 1 -- 172.21.15.150:0/3556243088 wait complete. 2024-04-25T14:02:20.324 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:02:20.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.437+0000 7f1b50996640 1 Processor -- start 2024-04-25T14:02:20.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.437+0000 7f1b50996640 1 -- start start 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b50996640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b50996640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d1000 0x7f1b4c06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b50996640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4c0d1750 con 0x7f1b4c160630 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b50996640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4c160a40 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b50996640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4c0d55d0 con 0x7f1b4c0d1000 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b4a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b4a59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47364/0 (socket says 172.21.15.150:47364) 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.438+0000 7f1b4a59c640 1 -- 172.21.15.150:0/3071032656 learned_addr learned my addr 172.21.15.150:0/3071032656 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:20.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4ad9d640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d1000 0x7f1b4c06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4a59c640 1 -- 172.21.15.150:0/3071032656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d1000 msgr2=0x7f1b4c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4a59c640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d1000 0x7f1b4c06c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4a59c640 1 -- 172.21.15.150:0/3071032656 >> v1:172.21.15.187:6789/0 conn(0x7f1b4c160630 legacy=0x7f1b4c0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4a59c640 1 -- 172.21.15.150:0/3071032656 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c0d4530 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.439+0000 7f1b4a59c640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009870 tx=0x7f1b3c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c36b5fc26f8e09 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b49d9b640 1 -- 172.21.15.150:0/3071032656 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c037070 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b49d9b640 1 -- 172.21.15.150:0/3071032656 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c032440 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b49d9b640 1 -- 172.21.15.150:0/3071032656 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c032810 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 -- 172.21.15.150:0/3071032656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0d3de0 msgr2=0x7f1b4c1600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c009870 tx=0x7f1b3c02a210 comp rx=0 tx=0).stop 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 -- 172.21.15.150:0/3071032656 shutdown_connections 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d1000 0x7f1b4c06c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 --2- 172.21.15.150:0/3071032656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0d3de0 0x7f1b4c1600f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 -- 172.21.15.150:0/3071032656 >> 172.21.15.150:0/3071032656 conn(0x7f1b4c0d25c0 msgr2=0x7f1b4c0d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:20.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 -- 172.21.15.150:0/3071032656 shutdown_connections 2024-04-25T14:02:20.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.440+0000 7f1b50996640 1 -- 172.21.15.150:0/3071032656 wait complete. 2024-04-25T14:02:20.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.442+0000 7f1b50996640 1 Processor -- start 2024-04-25T14:02:20.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.442+0000 7f1b50996640 1 -- start start 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d3de0 0x7f1b4c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 0x7f1b4c0cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c0d4dd0 con 0x7f1b4c0ce180 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c0d4af0 con 0x7f1b4c160630 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4c0d50b0 con 0x7f1b4c0d3de0 2024-04-25T14:02:20.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47378/0 (socket says 172.21.15.150:47378) 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 -- 172.21.15.150:0/3191004706 learned_addr learned my addr 172.21.15.150:0/3191004706 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4a59c640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d3de0 0x7f1b4c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b43fff640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 0x7f1b4c0cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 -- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d3de0 msgr2=0x7f1b4c0caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d3de0 0x7f1b4c0caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 -- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 msgr2=0x7f1b4c0cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 0x7f1b4c0cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 -- 172.21.15.150:0/3191004706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c159e40 con 0x7f1b4c0ce180 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b43fff640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 0x7f1b4c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b4ad9d640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b44007d80 tx=0x7f1b4400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44014420 con 0x7f1b4c0ce180 2024-04-25T14:02:20.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b44019070 con 0x7f1b4c0ce180 2024-04-25T14:02:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4c0c9a50 con 0x7f1b4c0ce180 2024-04-25T14:02:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.444+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b44014800 con 0x7f1b4c0ce180 2024-04-25T14:02:20.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.445+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4c0ca010 con 0x7f1b4c0ce180 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.445+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b4c076a40 con 0x7f1b4c0ce180 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.446+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1b440149a0 con 0x7f1b4c0ce180 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.446+0000 7f1b42ffd640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 0x7f1b2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.447+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1b44095cb0 con 0x7f1b4c0ce180 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.447+0000 7f1b4a59c640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 0x7f1b2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.447+0000 7f1b4a59c640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 0x7f1b2c0769c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c034000 tx=0x7f1b3c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.451+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b44060190 con 0x7f1b4c0ce180 2024-04-25T14:02:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.636+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1b4c0c97e0 con 0x7f1b4c0ce180 2024-04-25T14:02:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.637+0000 7f1b42ffd640 1 -- 172.21.15.150:0/3191004706 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7f1b4405fa90 con 0x7f1b4c0ce180 2024-04-25T14:02:20.637 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:02:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 msgr2=0x7f1b2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 0x7f1b2c0769c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c034000 tx=0x7f1b3c02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:20.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 msgr2=0x7f1b4c1f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1b44007d80 tx=0x7f1b4400e4b0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 shutdown_connections 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1b2c0744b0 0x7f1b2c0769c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b4c0ce180 0x7f1b4c1f5bf0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b4c160630 0x7f1b4c0cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 --2- 172.21.15.150:0/3191004706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b4c0d3de0 0x7f1b4c0caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 >> 172.21.15.150:0/3191004706 conn(0x7f1b4c0d25c0 msgr2=0x7f1b4c0d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 shutdown_connections 2024-04-25T14:02:20.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.639+0000 7f1b50996640 1 -- 172.21.15.150:0/3191004706 wait complete. 2024-04-25T14:02:20.650 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:20.651 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:20.666 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:20.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.813+0000 7f75f3b8f640 1 Processor -- start 2024-04-25T14:02:20.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.813+0000 7f75f3b8f640 1 -- start start 2024-04-25T14:02:20.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f3b8f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75ec0b8ec0 0x7f75ec0d1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f3b8f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75ec0b7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f3b8f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ec0d2910 con 0x7f75ec0d2100 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f3b8f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f75ec0b6970 con 0x7f75ec0b8ec0 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f3b8f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec0b6c50 con 0x7f75ec0d40e0 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f112a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f75ec0d2100 0x7f75ec0d39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49980/0 (socket says 172.21.15.150:49980) 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.814+0000 7f75f112a640 1 -- 172.21.15.150:0/1920259432 learned_addr learned my addr 172.21.15.150:0/1920259432 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75ec0b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f192b640 1 --2- 172.21.15.150:0/1920259432 >> v2:172.21.15.187:3300/0 conn(0x7f75ec0b8ec0 0x7f75ec0d1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1989636347 0 0) 0x7f75ec0d2910 con 0x7f75ec0d2100 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75d8003520 con 0x7f75ec0d2100 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3976957436 0 0) 0x7f75d8003520 con 0x7f75ec0d2100 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 msgr2=0x7f75ec0b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75ec0b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 >> v2:172.21.15.187:3300/0 conn(0x7f75ec0b8ec0 msgr2=0x7f75ec0d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 --2- 172.21.15.150:0/1920259432 >> v2:172.21.15.187:3300/0 conn(0x7f75ec0b8ec0 0x7f75ec0d1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec0b6f30 con 0x7f75ec0d2100 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f75dc001670 con 0x7f75ec0d2100 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 >> v1:172.21.15.150:6789/0 conn(0x7f75ec0d2100 legacy=0x7f75ec0d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75d80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d8003f30 0x7f75d8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75d8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec0b6970 con 0x7f75d8004d40 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec0b6c50 con 0x7f75d8003f30 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75d800ba10 con 0x7f75ec0d40e0 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75d8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f192b640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75d80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f112a640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d8003f30 0x7f75d8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 -- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 msgr2=0x7f75d80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0d40e0 0x7f75d80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 -- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d8003f30 msgr2=0x7f75d8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75d8003f30 0x7f75d8004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.815+0000 7f75f212c640 1 -- 172.21.15.150:0/1920259432 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75d8003520 con 0x7f75d8004d40 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.816+0000 7f75f212c640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75d8005150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0d1580 tx=0x7f75d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=164d4868fe6501e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.816+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4009c10 con 0x7f75d8004d40 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.816+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4036070 con 0x7f75d8004d40 2024-04-25T14:02:20.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.816+0000 7f75f0929640 1 -- 172.21.15.150:0/1920259432 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e4031830 con 0x7f75d8004d40 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1920259432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 msgr2=0x7f75d8005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1920259432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75d8005150 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f75ec0d1580 tx=0x7f75d8005830 comp rx=0 tx=0).stop 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f212c640 1 -- 172.21.15.150:0/1920259432 reap_dead start 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1920259432 shutdown_connections 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1920259432 >> 172.21.15.150:0/1920259432 conn(0x7f75ec08d5b0 msgr2=0x7f75ec09ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1920259432 shutdown_connections 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.817+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1920259432 wait complete. 2024-04-25T14:02:20.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.819+0000 7f75f3b8f640 1 Processor -- start 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.819+0000 7f75f3b8f640 1 -- start start 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75ec15f6d0 0x7f75ec14fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0b1ee0 0x7f75ec0b22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec0d2690 con 0x7f75d8004d40 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec0d2510 con 0x7f75ec15f6d0 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f3b8f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec0b7730 con 0x7f75ec0b1ee0 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f192b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f192b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47394/0 (socket says 172.21.15.150:47394) 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f192b640 1 -- 172.21.15.150:0/1079653760 learned_addr learned my addr 172.21.15.150:0/1079653760 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.820+0000 7f75f112a640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75ec15f6d0 0x7f75ec14fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f212c640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0b1ee0 0x7f75ec0b22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 -- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0b1ee0 msgr2=0x7f75ec0b22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0b1ee0 0x7f75ec0b22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 -- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75ec15f6d0 msgr2=0x7f75ec14fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75ec15f6d0 0x7f75ec14fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 -- 172.21.15.150:0/1079653760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec0b2830 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f192b640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011aa0 tx=0x7f75e8011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e800fb80 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e800fd20 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec15c740 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8014660 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.821+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec15e260 con 0x7f75d8004d40 2024-04-25T14:02:20.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.822+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ec076a40 con 0x7f75d8004d40 2024-04-25T14:02:20.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.823+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f75e8012040 con 0x7f75d8004d40 2024-04-25T14:02:20.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.823+0000 7f75e27fc640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 0x7f75d00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:20.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.823+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f75e8095c50 con 0x7f75d8004d40 2024-04-25T14:02:20.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.823+0000 7f75f112a640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 0x7f75d00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:20.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.824+0000 7f75f112a640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 0x7f75d00769c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75ec153b60 tx=0x7f75dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:20.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:20.828+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75e809b020 con 0x7f75d8004d40 2024-04-25T14:02:21.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.003+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75ec0c6fa0 con 0x7f75d8004d40 2024-04-25T14:02:21.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.004+0000 7f75e27fc640 1 -- 172.21.15.150:0/1079653760 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f75e8019030 con 0x7f75d8004d40 2024-04-25T14:02:21.004 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:21.005 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 msgr2=0x7f75d00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 0x7f75d00769c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f75ec153b60 tx=0x7f75dc009040 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 msgr2=0x7f75ec15f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f75e8011aa0 tx=0x7f75e8011f60 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 shutdown_connections 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f75d00744b0 0x7f75d00769c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75ec0b1ee0 0x7f75ec0b22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75ec15f6d0 0x7f75ec14fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 --2- 172.21.15.150:0/1079653760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75d8004d40 0x7f75ec15f020 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 >> 172.21.15.150:0/1079653760 conn(0x7f75ec08d5b0 msgr2=0x7f75ec0cfaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 shutdown_connections 2024-04-25T14:02:21.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.006+0000 7f75f3b8f640 1 -- 172.21.15.150:0/1079653760 wait complete. 2024-04-25T14:02:21.018 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:21.019 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:21.131 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.129+0000 7fd56a284640 1 Processor -- start 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.129+0000 7fd56a284640 1 -- start start 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd56a284640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5640cf3d0 0x7fd564070120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd56a284640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 0x7fd564070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd56a284640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5640d15d0 con 0x7fd56415e360 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd56a284640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd5640ce440 con 0x7fd5640cf3d0 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd56a284640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5640d1870 con 0x7fd5640d3080 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd563fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5640cf3d0 0x7fd564070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd563fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd5640cf3d0 0x7fd564070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47406/0 (socket says 172.21.15.150:47406) 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd563fff640 1 -- 172.21.15.150:0/872749405 learned_addr learned my addr 172.21.15.150:0/872749405 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.130+0000 7fd568821640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 0x7fd564070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd563fff640 1 -- 172.21.15.150:0/872749405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 msgr2=0x7fd564070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd563fff640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 0x7fd564070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd563fff640 1 -- 172.21.15.150:0/872749405 >> v1:172.21.15.187:6789/0 conn(0x7fd56415e360 legacy=0x7fd564161d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.132 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd563fff640 1 -- 172.21.15.150:0/872749405 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5640d1b50 con 0x7fd5640cf3d0 2024-04-25T14:02:21.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd568821640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 0x7fd564070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:21.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd563fff640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640cf3d0 0x7fd564070120 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd554009870 tx=0x7fd55402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28073e5947003d15 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd562ffd640 1 -- 172.21.15.150:0/872749405 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554037070 con 0x7fd5640cf3d0 2024-04-25T14:02:21.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd562ffd640 1 -- 172.21.15.150:0/872749405 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd554032440 con 0x7fd5640cf3d0 2024-04-25T14:02:21.133 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.131+0000 7fd562ffd640 1 -- 172.21.15.150:0/872749405 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd554032810 con 0x7fd5640cf3d0 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 -- 172.21.15.150:0/872749405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640cf3d0 msgr2=0x7fd564070120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640cf3d0 0x7fd564070120 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd554009870 tx=0x7fd55402a270 comp rx=0 tx=0).stop 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 -- 172.21.15.150:0/872749405 shutdown_connections 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd5640d3080 0x7fd564070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 --2- 172.21.15.150:0/872749405 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640cf3d0 0x7fd564070120 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 -- 172.21.15.150:0/872749405 >> 172.21.15.150:0/872749405 conn(0x7fd5640d3960 msgr2=0x7fd5640d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 -- 172.21.15.150:0/872749405 shutdown_connections 2024-04-25T14:02:21.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.132+0000 7fd56a284640 1 -- 172.21.15.150:0/872749405 wait complete. 2024-04-25T14:02:21.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.134+0000 7fd56a284640 1 Processor -- start 2024-04-25T14:02:21.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.134+0000 7fd56a284640 1 -- start start 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5640cf3d0 0x7fd5640ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd56415e360 0x7fd5640ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd564070eb0 con 0x7fd5640d3080 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd564070d30 con 0x7fd5640cf3d0 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.135+0000 7fd56a284640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd5640cdfb0 con 0x7fd56415e360 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47420/0 (socket says 172.21.15.150:47420) 2024-04-25T14:02:21.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 -- 172.21.15.150:0/3875124204 learned_addr learned my addr 172.21.15.150:0/3875124204 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd568821640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd56415e360 0x7fd5640ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd563fff640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5640cf3d0 0x7fd5640ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 -- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd56415e360 msgr2=0x7fd5640ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd56415e360 0x7fd5640ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 -- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5640cf3d0 msgr2=0x7fd5640ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5640cf3d0 0x7fd5640ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 -- 172.21.15.150:0/3875124204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd56415d910 con 0x7fd5640d3080 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd5637fe640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c3c0 tx=0x7fd55800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5580052d0 con 0x7fd5640d3080 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd558005490 con 0x7fd5640d3080 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5640ca200 con 0x7fd5640d3080 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd558007260 con 0x7fd5640d3080 2024-04-25T14:02:21.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.136+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd56415f4c0 con 0x7fd5640d3080 2024-04-25T14:02:21.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.137+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd564070b50 con 0x7fd5640d3080 2024-04-25T14:02:21.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.138+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd558007400 con 0x7fd5640d3080 2024-04-25T14:02:21.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.138+0000 7fd560ff9640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 0x7fd5500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.138+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd5580948e0 con 0x7fd5640d3080 2024-04-25T14:02:21.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.138+0000 7fd563fff640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 0x7fd5500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.139+0000 7fd563fff640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 0x7fd5500769c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd554034000 tx=0x7fd55402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.143+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd564070b50 con 0x7fd5640d3080 2024-04-25T14:02:21.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.324+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd564076a40 con 0x7fd5640d3080 2024-04-25T14:02:21.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.325+0000 7fd560ff9640 1 -- 172.21.15.150:0/3875124204 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7fd55805e770 con 0x7fd5640d3080 2024-04-25T14:02:21.325 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:21.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 msgr2=0x7fd5500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 0x7fd5500769c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd554034000 tx=0x7fd55402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:21.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 msgr2=0x7fd564153920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fd55800c3c0 tx=0x7fd55800c880 comp rx=0 tx=0).stop 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 shutdown_connections 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd5500744b0 0x7fd5500769c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd56415e360 0x7fd5640ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd5640d3080 0x7fd564153920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 --2- 172.21.15.150:0/3875124204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd5640cf3d0 0x7fd5640ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 >> 172.21.15.150:0/3875124204 conn(0x7fd5640d3960 msgr2=0x7fd5640b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 shutdown_connections 2024-04-25T14:02:21.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.327+0000 7fd56a284640 1 -- 172.21.15.150:0/3875124204 wait complete. 2024-04-25T14:02:21.338 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:21.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.449+0000 7facd1184640 1 Processor -- start 2024-04-25T14:02:21.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.449+0000 7facd1184640 1 -- start start 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.450+0000 7facd1184640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faccc0d3de0 0x7faccc1600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.450+0000 7facd1184640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faccc0d1000 0x7faccc06c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.450+0000 7facd1184640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faccc0d1750 con 0x7faccc160630 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.450+0000 7facd1184640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7faccc160a40 con 0x7faccc0d1000 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.450+0000 7facd1184640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faccc0d55d0 con 0x7faccc0d3de0 2024-04-25T14:02:21.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faccc0d1000 0x7faccc06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faccc0d1000 0x7faccc06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47436/0 (socket says 172.21.15.150:47436) 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7facca59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7faccc160630 0x7faccc0d08f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49982/0 (socket says 172.21.15.150:49982) 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 -- 172.21.15.150:0/3381695673 learned_addr learned my addr 172.21.15.150:0/3381695673 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccad9d640 1 --2- 172.21.15.150:0/3381695673 >> v2:172.21.15.187:3300/0 conn(0x7faccc0d3de0 0x7faccc1600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 -- 172.21.15.150:0/3381695673 >> v2:172.21.15.187:3300/0 conn(0x7faccc0d3de0 msgr2=0x7faccc1600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 --2- 172.21.15.150:0/3381695673 >> v2:172.21.15.187:3300/0 conn(0x7faccc0d3de0 0x7faccc1600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 -- 172.21.15.150:0/3381695673 >> v1:172.21.15.150:6789/0 conn(0x7faccc160630 legacy=0x7faccc0d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7faccb59e640 1 -- 172.21.15.150:0/3381695673 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc0d4530 con 0x7faccc0d1000 2024-04-25T14:02:21.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.451+0000 7facc9d9b640 1 -- 172.21.15.150:0/3381695673 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267835323 0 0) 0x7faccc0d1750 con 0x7faccc160630 2024-04-25T14:02:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7faccb59e640 1 --2- 172.21.15.150:0/3381695673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d1000 0x7faccc06c570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faccc08eb20 tx=0x7facc402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2643909da2c65a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facc9d9b640 1 -- 172.21.15.150:0/3381695673 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc4036070 con 0x7faccc0d1000 2024-04-25T14:02:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facc9d9b640 1 -- 172.21.15.150:0/3381695673 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc4031600 con 0x7faccc0d1000 2024-04-25T14:02:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facc9d9b640 1 -- 172.21.15.150:0/3381695673 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc40319d0 con 0x7faccc0d1000 2024-04-25T14:02:21.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 -- 172.21.15.150:0/3381695673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d1000 msgr2=0x7faccc06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 --2- 172.21.15.150:0/3381695673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d1000 0x7faccc06c570 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7faccc08eb20 tx=0x7facc402a730 comp rx=0 tx=0).stop 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 -- 172.21.15.150:0/3381695673 shutdown_connections 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 --2- 172.21.15.150:0/3381695673 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d1000 0x7faccc06c570 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 --2- 172.21.15.150:0/3381695673 >> v2:172.21.15.187:3300/0 conn(0x7faccc0d3de0 0x7faccc1600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.452+0000 7facd1184640 1 -- 172.21.15.150:0/3381695673 >> 172.21.15.150:0/3381695673 conn(0x7faccc0d25c0 msgr2=0x7faccc0d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.453+0000 7facd1184640 1 -- 172.21.15.150:0/3381695673 shutdown_connections 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.453+0000 7facd1184640 1 -- 172.21.15.150:0/3381695673 wait complete. 2024-04-25T14:02:21.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.455+0000 7facd1184640 1 Processor -- start 2024-04-25T14:02:21.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.455+0000 7facd1184640 1 -- start start 2024-04-25T14:02:21.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 0x7faccc1f9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faccc162b00 0x7faccc162f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faccc0d4dd0 con 0x7faccc0d3de0 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faccc0d4af0 con 0x7faccc160630 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facd1184640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faccc0d50b0 con 0x7faccc162b00 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47444/0 (socket says 172.21.15.150:47444) 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 -- 172.21.15.150:0/2734772059 learned_addr learned my addr 172.21.15.150:0/2734772059 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccb59e640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faccc162b00 0x7faccc162f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facca59c640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 0x7faccc1f9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 -- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faccc162b00 msgr2=0x7faccc162f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faccc162b00 0x7faccc162f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 -- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 msgr2=0x7faccc1f9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 0x7faccc1f9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 -- 172.21.15.150:0/2734772059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faccc153b40 con 0x7faccc0d3de0 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7facca59c640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 0x7faccc1f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:21.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.456+0000 7faccad9d640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e870 tx=0x7facbc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.457+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc017070 con 0x7faccc0d3de0 2024-04-25T14:02:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.457+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facbc00c940 con 0x7faccc0d3de0 2024-04-25T14:02:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.457+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faccc163450 con 0x7faccc0d3de0 2024-04-25T14:02:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.457+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc00cd10 con 0x7faccc0d3de0 2024-04-25T14:02:21.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.457+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faccc156390 con 0x7faccc0d3de0 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.458+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faccc0697d0 con 0x7faccc0d3de0 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.458+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7facbc01a070 con 0x7faccc0d3de0 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.459+0000 7facb37fe640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 0x7facb80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.459+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7facbc0953c0 con 0x7faccc0d3de0 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.459+0000 7facca59c640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 0x7facb80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.459+0000 7facca59c640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 0x7facb80769c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7facc0000f60 tx=0x7facc0007cc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.463+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faccc0697d0 con 0x7faccc0d3de0 2024-04-25T14:02:21.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.638+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7faccc076a40 con 0x7faccc0d3de0 2024-04-25T14:02:21.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.638+0000 7facb37fe640 1 -- 172.21.15.150:0/2734772059 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7faccc163450 con 0x7faccc0d3de0 2024-04-25T14:02:21.639 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 msgr2=0x7facb80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 0x7facb80769c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7facc0000f60 tx=0x7facc0007cc0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 msgr2=0x7faccc1f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7facbc00e870 tx=0x7facbc00ed30 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 shutdown_connections 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb80744b0 0x7facb80769c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faccc162b00 0x7faccc162f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faccc160630 0x7faccc1f9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.640+0000 7facd1184640 1 --2- 172.21.15.150:0/2734772059 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faccc0d3de0 0x7faccc1f5660 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.641+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 >> 172.21.15.150:0/2734772059 conn(0x7faccc0d25c0 msgr2=0x7faccc15db90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.641+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 shutdown_connections 2024-04-25T14:02:21.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.641+0000 7facd1184640 1 -- 172.21.15.150:0/2734772059 wait complete. 2024-04-25T14:02:21.652 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:21.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.762+0000 7f608b8b1640 1 Processor -- start 2024-04-25T14:02:21.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.762+0000 7f608b8b1640 1 -- start start 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608b8b1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f60840d4fc0 0x7f6084160330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608b8b1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608406ca50 con 0x7f608406c570 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608b8b1640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f608406cd30 con 0x7f608406daf0 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608b8b1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f608406df00 con 0x7f60840d4fc0 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608964d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f608406c570 0x7f60840d3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:49998/0 (socket says 172.21.15.150:49998) 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f608964d640 1 -- 172.21.15.150:0/2232943020 learned_addr learned my addr 172.21.15.150:0/2232943020 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f6089e4e640 1 --2- 172.21.15.150:0/2232943020 >> v2:172.21.15.187:3300/0 conn(0x7f60840d4fc0 0x7f6084160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2984602507 0 0) 0x7f608406ca50 con 0x7f608406c570 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6070003520 con 0x7f608406c570 2024-04-25T14:02:21.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2753119354 0 0) 0x7f6070003520 con 0x7f608406c570 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 >> v2:172.21.15.187:3300/0 conn(0x7f60840d4fc0 msgr2=0x7f6084160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 --2- 172.21.15.150:0/2232943020 >> v2:172.21.15.187:3300/0 conn(0x7f60840d4fc0 0x7f6084160330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 >> v1:172.21.15.187:6789/0 conn(0x7f608406daf0 legacy=0x7f60840d48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f608406d010 con 0x7f608406c570 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f6080001670 con 0x7f608406c570 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 >> v1:172.21.15.150:6789/0 conn(0x7f608406c570 legacy=0x7f60840d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 0x7f6070003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6070004800 0x7f6070004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6070005150 0x7f607000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f608406df00 con 0x7f60700039b0 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f607000c0c0 con 0x7f6070004800 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.764+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f607000c2c0 con 0x7f6070005150 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 0x7f6070003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f6088e4c640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6070004800 0x7f6070004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f6089e4e640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6070005150 0x7f607000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 -- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6070005150 msgr2=0x7f607000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6070005150 0x7f607000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 -- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6070004800 msgr2=0x7f6070004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6070004800 0x7f6070004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f6089e4e640 1 -- 172.21.15.150:0/2232943020 reap_dead start 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 -- 172.21.15.150:0/2232943020 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6070003520 con 0x7f60700039b0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.765+0000 7f608964d640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 0x7f6070003de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f608000c240 tx=0x7f6070005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23d590f8a4ef0ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080007d30 con 0x7f60700039b0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6080005af0 con 0x7f60700039b0 2024-04-25T14:02:21.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f607bfff640 1 -- 172.21.15.150:0/2232943020 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6080007070 con 0x7f60700039b0 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 -- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 msgr2=0x7f6070003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 0x7f6070003de0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f608000c240 tx=0x7f6070005830 comp rx=0 tx=0).stop 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 -- 172.21.15.150:0/2232943020 shutdown_connections 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 --2- 172.21.15.150:0/2232943020 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60700039b0 0x7f6070003de0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 -- 172.21.15.150:0/2232943020 >> 172.21.15.150:0/2232943020 conn(0x7f60840d63f0 msgr2=0x7f60840d2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 -- 172.21.15.150:0/2232943020 shutdown_connections 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.766+0000 7f608b8b1640 1 -- 172.21.15.150:0/2232943020 wait complete. 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.768+0000 7f608b8b1640 1 Processor -- start 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.768+0000 7f608b8b1640 1 -- start start 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60700039b0 0x7f608415c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.771 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60840ce1f0 0x7f60841608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60841604b0 con 0x7f60840cc410 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6084160330 con 0x7f60840ce1f0 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.769+0000 7f608b8b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6084160630 con 0x7f60700039b0 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47458/0 (socket says 172.21.15.150:47458) 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 -- 172.21.15.150:0/4010603315 learned_addr learned my addr 172.21.15.150:0/4010603315 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6089e4e640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60840ce1f0 0x7f60841608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f608964d640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60700039b0 0x7f608415c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 -- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60700039b0 msgr2=0x7f608415c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60700039b0 0x7f608415c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 -- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60840ce1f0 msgr2=0x7f60841608a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60840ce1f0 0x7f60841608a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 -- 172.21.15.150:0/4010603315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60840ce600 con 0x7f60840cc410 2024-04-25T14:02:21.772 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6088e4c640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f607400f7d0 tx=0x7f607400fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607400fe80 con 0x7f60840cc410 2024-04-25T14:02:21.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6074018070 con 0x7f60840cc410 2024-04-25T14:02:21.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6084161a00 con 0x7f60840cc410 2024-04-25T14:02:21.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f607400d6b0 con 0x7f60840cc410 2024-04-25T14:02:21.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.770+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60840ad3d0 con 0x7f60840cc410 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.771+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6084076a40 con 0x7f60840cc410 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.772+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f607400d850 con 0x7f60840cc410 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.772+0000 7f6079ffb640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 0x7f60680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.772+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f60740952d0 con 0x7f60840cc410 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.772+0000 7f608964d640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 0x7f60680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:21.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.773+0000 7f608964d640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 0x7f60680769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6080032000 tx=0x7f608002d000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:21.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.777+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f60840ce600 con 0x7f60840cc410 2024-04-25T14:02:21.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.952+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f60840d14a0 con 0x7f60840cc410 2024-04-25T14:02:21.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.952+0000 7f6079ffb640 1 -- 172.21.15.150:0/4010603315 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f60740950f0 con 0x7f60840cc410 2024-04-25T14:02:21.952 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 msgr2=0x7f60680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 0x7f60680769f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6080032000 tx=0x7f608002d000 comp rx=0 tx=0).stop 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 msgr2=0x7f60840cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f607400f7d0 tx=0x7f607400fc90 comp rx=0 tx=0).stop 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 shutdown_connections 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60680744e0 0x7f60680769f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f60840ce1f0 0x7f60841608a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f60840cc410 0x7f60840cdcb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 --2- 172.21.15.150:0/4010603315 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f60700039b0 0x7f608415c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 >> 172.21.15.150:0/4010603315 conn(0x7f60840d63f0 msgr2=0x7f60840d48b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 shutdown_connections 2024-04-25T14:02:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:21.954+0000 7f608b8b1640 1 -- 172.21.15.150:0/4010603315 wait complete. 2024-04-25T14:02:21.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:22.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.077+0000 7f29c786e640 1 Processor -- start 2024-04-25T14:02:22.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.077+0000 7f29c786e640 1 -- start start 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c786e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29c0163d80 0x7f29c006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c786e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c006ff50 con 0x7f29c0160460 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c786e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c006f3d0 con 0x7f29c00b5ad0 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c786e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f29c006f6b0 con 0x7f29c0163d80 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c560a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f29c0160460 0x7f29c00b53c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:50000/0 (socket says 172.21.15.150:50000) 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c560a640 1 -- 172.21.15.150:0/976907487 learned_addr learned my addr 172.21.15.150:0/976907487 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c5e0b640 1 --2- 172.21.15.150:0/976907487 >> v2:172.21.15.187:3300/0 conn(0x7f29c0163d80 0x7f29c006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29b7fff640 1 -- 172.21.15.150:0/976907487 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309831391 0 0) 0x7f29c006ff50 con 0x7f29c0160460 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29b7fff640 1 -- 172.21.15.150:0/976907487 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29ac003520 con 0x7f29c0160460 2024-04-25T14:02:22.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c5e0b640 1 -- 172.21.15.150:0/976907487 >> v1:172.21.15.187:6789/0 conn(0x7f29c00b5ad0 legacy=0x7f29c00d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c5e0b640 1 -- 172.21.15.150:0/976907487 >> v1:172.21.15.150:6789/0 conn(0x7f29c0160460 legacy=0x7f29c00b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.079+0000 7f29c5e0b640 1 -- 172.21.15.150:0/976907487 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c006f990 con 0x7f29c0163d80 2024-04-25T14:02:22.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.080+0000 7f29c5e0b640 1 --2- 172.21.15.150:0/976907487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c006ed40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=794b889536bbc061 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.080+0000 7f29b7fff640 1 -- 172.21.15.150:0/976907487 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8007740 con 0x7f29c0163d80 2024-04-25T14:02:22.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.080+0000 7f29b7fff640 1 -- 172.21.15.150:0/976907487 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b80078e0 con 0x7f29c0163d80 2024-04-25T14:02:22.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.080+0000 7f29b7fff640 1 -- 172.21.15.150:0/976907487 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8007c90 con 0x7f29c0163d80 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 -- 172.21.15.150:0/976907487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 msgr2=0x7f29c006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 --2- 172.21.15.150:0/976907487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c006ed40 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b802a660 comp rx=0 tx=0).stop 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 -- 172.21.15.150:0/976907487 shutdown_connections 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 --2- 172.21.15.150:0/976907487 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c006ed40 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 -- 172.21.15.150:0/976907487 >> 172.21.15.150:0/976907487 conn(0x7f29c00d03c0 msgr2=0x7f29c00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 -- 172.21.15.150:0/976907487 shutdown_connections 2024-04-25T14:02:22.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.081+0000 7f29c786e640 1 -- 172.21.15.150:0/976907487 wait complete. 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.083+0000 7f29c786e640 1 Processor -- start 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.083+0000 7f29c786e640 1 -- start start 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c00b67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c00b8220 0x7f29c00affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29c00ce5c0 con 0x7f29c00b04e0 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f29c00b3030 con 0x7f29c0163d80 2024-04-25T14:02:22.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c786e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29c00d1f70 con 0x7f29c00b8220 2024-04-25T14:02:22.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47472/0 (socket says 172.21.15.150:47472) 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 -- 172.21.15.150:0/1913194748 learned_addr learned my addr 172.21.15.150:0/1913194748 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c560a640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c00b67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c4e09640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c00b8220 0x7f29c00affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 -- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c00b8220 msgr2=0x7f29c00affa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c00b8220 0x7f29c00affa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 -- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 msgr2=0x7f29c00b67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c00b67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.084+0000 7f29c5e0b640 1 -- 172.21.15.150:0/1913194748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c00b7180 con 0x7f29c00b04e0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29c5e0b640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b803f070 con 0x7f29c00b04e0 2024-04-25T14:02:22.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b8031c40 con 0x7f29c00b04e0 2024-04-25T14:02:22.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8009450 con 0x7f29c00b04e0 2024-04-25T14:02:22.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c00b7460 con 0x7f29c00b04e0 2024-04-25T14:02:22.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.085+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c00b7a20 con 0x7f29c00b04e0 2024-04-25T14:02:22.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.086+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c00b6ef0 con 0x7f29c00b04e0 2024-04-25T14:02:22.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.087+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f29b8003990 con 0x7f29c00b04e0 2024-04-25T14:02:22.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.087+0000 7f29b5ffb640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 0x7f29a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.087+0000 7f29c560a640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 0x7f29a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.087+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f29b80b6300 con 0x7f29c00b04e0 2024-04-25T14:02:22.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.088+0000 7f29c560a640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 0x7f29a40769c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0011e0 tx=0x7f29bc007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.092+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29b8080110 con 0x7f29c00b04e0 2024-04-25T14:02:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.266+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f29c0076a40 con 0x7f29c00b04e0 2024-04-25T14:02:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.267+0000 7f29b5ffb640 1 -- 172.21.15.150:0/1913194748 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f29b8084410 con 0x7f29c00b04e0 2024-04-25T14:02:22.267 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 msgr2=0x7f29a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 0x7f29a40769c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f29bc0011e0 tx=0x7f29bc007d30 comp rx=0 tx=0).stop 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 msgr2=0x7f29c01614b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f29b8009d90 tx=0x7f29b8004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 shutdown_connections 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29a40744b0 0x7f29a40769c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c00b04e0 0x7f29c01614b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c00b8220 0x7f29c00affa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 --2- 172.21.15.150:0/1913194748 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c0163d80 0x7f29c00b67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 >> 172.21.15.150:0/1913194748 conn(0x7f29c00d03c0 msgr2=0x7f29c00d22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 shutdown_connections 2024-04-25T14:02:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.269+0000 7f29c786e640 1 -- 172.21.15.150:0/1913194748 wait complete. 2024-04-25T14:02:22.280 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:02:22.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.389+0000 7ff25e261640 1 Processor -- start 2024-04-25T14:02:22.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.389+0000 7ff25e261640 1 -- start start 2024-04-25T14:02:22.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff25e261640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2580b4c10 0x7ff258160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff25e261640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff25806d800 0x7ff2580d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff25e261640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2580b9600 con 0x7ff2580b91f0 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff25e261640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff25806cc60 con 0x7ff2580b4c10 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff25e261640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff2580d4f20 con 0x7ff25806d800 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2580b4c10 0x7ff258160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff2580b4c10 0x7ff258160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47486/0 (socket says 172.21.15.150:47486) 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 -- 172.21.15.150:0/63608874 learned_addr learned my addr 172.21.15.150:0/63608874 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff257fff640 1 --2- 172.21.15.150:0/63608874 >> v2:172.21.15.187:3300/0 conn(0x7ff25806d800 0x7ff2580d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 -- 172.21.15.150:0/63608874 >> v2:172.21.15.187:3300/0 conn(0x7ff25806d800 msgr2=0x7ff2580d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 --2- 172.21.15.150:0/63608874 >> v2:172.21.15.187:3300/0 conn(0x7ff25806d800 0x7ff2580d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 -- 172.21.15.150:0/63608874 >> v1:172.21.15.187:6789/0 conn(0x7ff2580b91f0 legacy=0x7ff25806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.391+0000 7ff2577fe640 1 -- 172.21.15.150:0/63608874 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2580d5200 con 0x7ff2580b4c10 2024-04-25T14:02:22.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.392+0000 7ff2577fe640 1 --2- 172.21.15.150:0/63608874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2580b4c10 0x7ff258160150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff2580d0690 tx=0x7ff24802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd7778e987928c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.392+0000 7ff2567fc640 1 -- 172.21.15.150:0/63608874 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248037070 con 0x7ff2580b4c10 2024-04-25T14:02:22.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.392+0000 7ff2567fc640 1 -- 172.21.15.150:0/63608874 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff248032440 con 0x7ff2580b4c10 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.392+0000 7ff2567fc640 1 -- 172.21.15.150:0/63608874 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248032810 con 0x7ff2580b4c10 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 -- 172.21.15.150:0/63608874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2580b4c10 msgr2=0x7ff258160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 --2- 172.21.15.150:0/63608874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2580b4c10 0x7ff258160150 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff2580d0690 tx=0x7ff24802a270 comp rx=0 tx=0).stop 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 -- 172.21.15.150:0/63608874 shutdown_connections 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 --2- 172.21.15.150:0/63608874 >> v2:172.21.15.187:3300/0 conn(0x7ff25806d800 0x7ff2580d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 --2- 172.21.15.150:0/63608874 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2580b4c10 0x7ff258160150 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 -- 172.21.15.150:0/63608874 >> 172.21.15.150:0/63608874 conn(0x7ff2580b7690 msgr2=0x7ff2580b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 -- 172.21.15.150:0/63608874 shutdown_connections 2024-04-25T14:02:22.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.393+0000 7ff25e261640 1 -- 172.21.15.150:0/63608874 wait complete. 2024-04-25T14:02:22.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.395+0000 7ff25e261640 1 Processor -- start 2024-04-25T14:02:22.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.395+0000 7ff25e261640 1 -- start start 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2580b4c10 0x7ff2581f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 0x7ff2580c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff25806c8f0 con 0x7ff25806d800 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff25806c610 con 0x7ff2580b4c10 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff25e261640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff25806dc10 con 0x7ff2580b91f0 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47490/0 (socket says 172.21.15.150:47490) 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 -- 172.21.15.150:0/380491274 learned_addr learned my addr 172.21.15.150:0/380491274 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff256ffd640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2580b4c10 0x7ff2581f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff257fff640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 0x7ff2580c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 -- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 msgr2=0x7ff2580c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 0x7ff2580c9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 -- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2580b4c10 msgr2=0x7ff2581f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2580b4c10 0x7ff2581f61f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.396+0000 7ff2577fe640 1 -- 172.21.15.150:0/380491274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff25814e5a0 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff257fff640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 0x7ff2580c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff2577fe640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff248000c00 tx=0x7ff2480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248041020 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2480306c0 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff25815d0d0 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff248037040 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.397+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff25814bef0 con 0x7ff25806d800 2024-04-25T14:02:22.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.398+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff258076a40 con 0x7ff25806d800 2024-04-25T14:02:22.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.399+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff248044020 con 0x7ff25806d800 2024-04-25T14:02:22.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.399+0000 7ff23ffff640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 0x7ff2440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:22.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.399+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff2480b5d60 con 0x7ff25806d800 2024-04-25T14:02:22.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.399+0000 7ff256ffd640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 0x7ff2440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:22.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.399+0000 7ff256ffd640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 0x7ff2440769f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff25814ed30 tx=0x7ff24c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:22.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.404+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff24804b030 con 0x7ff25806d800 2024-04-25T14:02:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.592+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff2580b7470 con 0x7ff25806d800 2024-04-25T14:02:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.592+0000 7ff23ffff640 1 -- 172.21.15.150:0/380491274 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) ==== 80+0+0 (secure 0 0 0) 0x7ff2480801b0 con 0x7ff25806d800 2024-04-25T14:02:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 msgr2=0x7ff2440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 0x7ff2440769f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff25814ed30 tx=0x7ff24c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 msgr2=0x7ff25815c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff248000c00 tx=0x7ff2480046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 shutdown_connections 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2440744e0 0x7ff2440769f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2580b91f0 0x7ff2580c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2580b4c10 0x7ff2581f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 --2- 172.21.15.150:0/380491274 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff25806d800 0x7ff25815c580 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.594+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 >> 172.21.15.150:0/380491274 conn(0x7ff2580b7690 msgr2=0x7ff2580ba590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.595+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 shutdown_connections 2024-04-25T14:02:22.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:22.595+0000 7ff25e261640 1 -- 172.21.15.150:0/380491274 wait complete. 2024-04-25T14:02:22.607 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi150.front.sepia.ceph.com... 2024-04-25T14:02:22.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:22.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-25T14:02:22.629 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi150.stderr:ceph-fuse[42164]: fuse finished with error 0 and tester_r 0 2024-04-25T14:02:22.631 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:02:28.733 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:02:28.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:28.733 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-25T14:02:28.766 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:28.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.924+0000 7f81af386640 1 Processor -- start 2024-04-25T14:02:28.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.925+0000 7f81af386640 1 -- start start 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81af386640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81a8160210 0x7f81a80d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81af386640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 0x7f81a80cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81af386640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81a80d20a0 con 0x7f81a80b2ba0 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81af386640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f81a80d2380 con 0x7f81a8160210 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81af386640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81a80cfc30 con 0x7f81a80b5ab0 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad122640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f81a8160210 0x7f81a80d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ac921640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f81a80b2ba0 0x7f81a80b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55438/0 (socket says 172.21.15.150:55438) 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ac921640 1 -- 172.21.15.150:0/2015733417 learned_addr learned my addr 172.21.15.150:0/2015733417 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad923640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 0x7f81a80cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad122640 1 -- 172.21.15.150:0/2015733417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 msgr2=0x7f81a80cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:28.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad122640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 0x7f81a80cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:28.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad122640 1 -- 172.21.15.150:0/2015733417 >> v1:172.21.15.150:6789/0 conn(0x7f81a80b2ba0 legacy=0x7f81a80b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:28.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad122640 1 -- 172.21.15.150:0/2015733417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a80d0150 con 0x7f81a8160210 2024-04-25T14:02:28.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f819ffff640 1 -- 172.21.15.150:0/2015733417 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4268859634 0 0) 0x7f81a80d20a0 con 0x7f81a80b2ba0 2024-04-25T14:02:28.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.926+0000 7f81ad923640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 0x7f81a80cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.927+0000 7f81ad122640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a8160210 0x7f81a80d2bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81a40097a0 tx=0x7f81a402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11fc001127159e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.927+0000 7f819ffff640 1 -- 172.21.15.150:0/2015733417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4037070 con 0x7f81a8160210 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.927+0000 7f819ffff640 1 -- 172.21.15.150:0/2015733417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a4032440 con 0x7f81a8160210 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.927+0000 7f819ffff640 1 -- 172.21.15.150:0/2015733417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a4032810 con 0x7f81a8160210 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 -- 172.21.15.150:0/2015733417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a8160210 msgr2=0x7f81a80d2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a8160210 0x7f81a80d2bd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f81a40097a0 tx=0x7f81a402a180 comp rx=0 tx=0).stop 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 -- 172.21.15.150:0/2015733417 shutdown_connections 2024-04-25T14:02:28.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b5ab0 0x7f81a80cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 --2- 172.21.15.150:0/2015733417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a8160210 0x7f81a80d2bd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 -- 172.21.15.150:0/2015733417 >> 172.21.15.150:0/2015733417 conn(0x7f81a80d03a0 msgr2=0x7f81a80d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 -- 172.21.15.150:0/2015733417 shutdown_connections 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.928+0000 7f81af386640 1 -- 172.21.15.150:0/2015733417 wait complete. 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.930+0000 7f81af386640 1 Processor -- start 2024-04-25T14:02:28.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.930+0000 7f81af386640 1 -- start start 2024-04-25T14:02:28.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b2ba0 0x7f81a806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81a8160210 0x7f81a815aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f81a80ce630 con 0x7f81a80c9ab0 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f81a80ce4b0 con 0x7f81a8160210 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81af386640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f81a80ce7b0 con 0x7f81a80b2ba0 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81ad923640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81ad923640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38136/0 (socket says 172.21.15.150:38136) 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.931+0000 7f81ad923640 1 -- 172.21.15.150:0/1454921118 learned_addr learned my addr 172.21.15.150:0/1454921118 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad122640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b2ba0 0x7f81a806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ac921640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81a8160210 0x7f81a815aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 -- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b2ba0 msgr2=0x7f81a806eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b2ba0 0x7f81a806eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 -- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81a8160210 msgr2=0x7f81a815aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81a8160210 0x7f81a815aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:28.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 -- 172.21.15.150:0/1454921118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81a80c9220 con 0x7f81a80c9ab0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81ad923640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f81a000cd20 tx=0x7f81a000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a000f680 con 0x7f81a80c9ab0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81a000f840 con 0x7f81a80c9ab0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81a806e0b0 con 0x7f81a80c9ab0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81a000fc30 con 0x7f81a80c9ab0 2024-04-25T14:02:28.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.932+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81a806e670 con 0x7f81a80c9ab0 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.934+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81a8076a40 con 0x7f81a80c9ab0 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.934+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f81a00213f0 con 0x7f81a80c9ab0 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.934+0000 7f819dffb640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 0x7f818c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.935+0000 7f81ad122640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 0x7f818c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.935+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f81a00961d0 con 0x7f81a80c9ab0 2024-04-25T14:02:28.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.935+0000 7f81ad122640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 0x7f818c0769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81a4034000 tx=0x7f81a402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:28.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:28.939+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f81a0014030 con 0x7f81a80c9ab0 2024-04-25T14:02:29.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.116+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81a80c8fb0 con 0x7f81a80c9ab0 2024-04-25T14:02:29.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.117+0000 7f819dffb640 1 -- 172.21.15.150:0/1454921118 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f81a005ffd0 con 0x7f81a80c9ab0 2024-04-25T14:02:29.118 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:29.118 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 msgr2=0x7f818c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 0x7f818c0769c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f81a4034000 tx=0x7f81a402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 msgr2=0x7f81a80ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f81a000cd20 tx=0x7f81a000a650 comp rx=0 tx=0).stop 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 shutdown_connections 2024-04-25T14:02:29.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f818c0744b0 0x7f818c0769c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f81a80c9ab0 0x7f81a80ccb60 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f81a8160210 0x7f81a815aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 --2- 172.21.15.150:0/1454921118 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f81a80b2ba0 0x7f81a806eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 >> 172.21.15.150:0/1454921118 conn(0x7f81a80d03a0 msgr2=0x7f81a80ceb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 shutdown_connections 2024-04-25T14:02:29.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.120+0000 7f81af386640 1 -- 172.21.15.150:0/1454921118 wait complete. 2024-04-25T14:02:29.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:29.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.242+0000 7fd828884640 1 Processor -- start 2024-04-25T14:02:29.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.242+0000 7fd828884640 1 -- start start 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.243+0000 7fd828884640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8240b4e00 0x7fd8240d4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.243+0000 7fd828884640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 0x7fd8240b9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.243+0000 7fd828884640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd82406d790 con 0x7fd82406cf80 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.243+0000 7fd828884640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd8240b9680 con 0x7fd8240b4e00 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.243+0000 7fd828884640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd82406da70 con 0x7fd8240d1290 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8240b4e00 0x7fd8240d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd8240b4e00 0x7fd8240d4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38150/0 (socket says 172.21.15.150:38150) 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 -- 172.21.15.150:0/1995144987 learned_addr learned my addr 172.21.15.150:0/1995144987 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd822d9d640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 0x7fd8240b9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 -- 172.21.15.150:0/1995144987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 msgr2=0x7fd8240b9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 0x7fd8240b9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 -- 172.21.15.150:0/1995144987 >> v1:172.21.15.187:6789/0 conn(0x7fd82406cf80 legacy=0x7fd8240d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd82259c640 1 -- 172.21.15.150:0/1995144987 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82406dd50 con 0x7fd8240b4e00 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.244+0000 7fd822d9d640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 0x7fd8240b9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd82259c640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8240b4e00 0x7fd8240d4200 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd82408eaf0 tx=0x7fd81402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532bb1606667034e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd82159a640 1 -- 172.21.15.150:0/1995144987 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd814036070 con 0x7fd8240b4e00 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd82159a640 1 -- 172.21.15.150:0/1995144987 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd814031600 con 0x7fd8240b4e00 2024-04-25T14:02:29.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd82159a640 1 -- 172.21.15.150:0/1995144987 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8140319d0 con 0x7fd8240b4e00 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd828884640 1 -- 172.21.15.150:0/1995144987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8240b4e00 msgr2=0x7fd8240d4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.245+0000 7fd828884640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8240b4e00 0x7fd8240d4200 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd82408eaf0 tx=0x7fd81402a730 comp rx=0 tx=0).stop 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 -- 172.21.15.150:0/1995144987 shutdown_connections 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd8240d1290 0x7fd8240b9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 --2- 172.21.15.150:0/1995144987 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd8240b4e00 0x7fd8240d4200 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 -- 172.21.15.150:0/1995144987 >> 172.21.15.150:0/1995144987 conn(0x7fd8240d63d0 msgr2=0x7fd8240d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 -- 172.21.15.150:0/1995144987 shutdown_connections 2024-04-25T14:02:29.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.246+0000 7fd828884640 1 -- 172.21.15.150:0/1995144987 wait complete. 2024-04-25T14:02:29.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.247+0000 7fd828884640 1 Processor -- start 2024-04-25T14:02:29.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.247+0000 7fd828884640 1 -- start start 2024-04-25T14:02:29.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd82406cf80 0x7fd82415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8240b4e00 0x7fd82414ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd82406d510 con 0x7fd824159340 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd82406d390 con 0x7fd8240b4e00 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd828884640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd8240ba050 con 0x7fd82406cf80 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38156/0 (socket says 172.21.15.150:38156) 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 -- 172.21.15.150:0/2993292737 learned_addr learned my addr 172.21.15.150:0/2993292737 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd82259c640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd82406cf80 0x7fd82415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd821d9b640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8240b4e00 0x7fd82414ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 -- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd82406cf80 msgr2=0x7fd82415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd82406cf80 0x7fd82415c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 -- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8240b4e00 msgr2=0x7fd82414ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8240b4e00 0x7fd82414ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 -- 172.21.15.150:0/2993292737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82415c920 con 0x7fd824159340 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.249+0000 7fd822d9d640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00c5c0 tx=0x7fd81c00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.250+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c015070 con 0x7fd824159340 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.250+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd81c011820 con 0x7fd824159340 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.250+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd824145320 con 0x7fd824159340 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.250+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c011c10 con 0x7fd824159340 2024-04-25T14:02:29.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.250+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8241458e0 con 0x7fd824159340 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.251+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd824076a40 con 0x7fd824159340 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.251+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd81c018070 con 0x7fd824159340 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.252+0000 7fd812ffd640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 0x7fd8040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.252+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd81c095390 con 0x7fd824159340 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.252+0000 7fd82259c640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 0x7fd8040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.252+0000 7fd82259c640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 0x7fd8040769c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd814033000 tx=0x7fd814009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.256+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd81c05f1a0 con 0x7fd824159340 2024-04-25T14:02:29.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.432+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd82415c670 con 0x7fd824159340 2024-04-25T14:02:29.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.433+0000 7fd812ffd640 1 -- 172.21.15.150:0/2993292737 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fd81c0634a0 con 0x7fd824159340 2024-04-25T14:02:29.434 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:29.434 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 msgr2=0x7fd8040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 0x7fd8040769c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd814033000 tx=0x7fd814009000 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 msgr2=0x7fd8240ba8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00c5c0 tx=0x7fd81c00ca80 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 shutdown_connections 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd8040744b0 0x7fd8040769c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd824159340 0x7fd8240ba8e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd8240b4e00 0x7fd82414ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 --2- 172.21.15.150:0/2993292737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd82406cf80 0x7fd82415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 >> 172.21.15.150:0/2993292737 conn(0x7fd8240d63d0 msgr2=0x7fd8240684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 shutdown_connections 2024-04-25T14:02:29.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.435+0000 7fd828884640 1 -- 172.21.15.150:0/2993292737 wait complete. 2024-04-25T14:02:29.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T14:02:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.557+0000 7f645a597640 1 Processor -- start 2024-04-25T14:02:29.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.557+0000 7f645a597640 1 -- start start 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f645a597640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f64540b4260 0x7f64540cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f645a597640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f64540ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f645a597640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64540ce9d0 con 0x7f64540d1bc0 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f645a597640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f64540cf5e0 con 0x7f64540b4260 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f645a597640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64540cecb0 con 0x7f64540ce4d0 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6453fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f64540d1bc0 0x7f64540b3b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55444/0 (socket says 172.21.15.150:55444) 2024-04-25T14:02:29.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6453fff640 1 -- 172.21.15.150:0/2974572834 learned_addr learned my addr 172.21.15.150:0/2974572834 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f64537fe640 1 --2- 172.21.15.150:0/2974572834 >> v2:172.21.15.187:3300/0 conn(0x7f64540b4260 0x7f64540cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6458b34640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f64540ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157655903 0 0) 0x7f64540ce9d0 con 0x7f64540d1bc0 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f643c003520 con 0x7f64540d1bc0 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3147287896 0 0) 0x7f643c003520 con 0x7f64540d1bc0 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 msgr2=0x7f64540ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f64540ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 >> v2:172.21.15.187:3300/0 conn(0x7f64540b4260 msgr2=0x7f64540cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 --2- 172.21.15.150:0/2974572834 >> v2:172.21.15.187:3300/0 conn(0x7f64540b4260 0x7f64540cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64540cef90 con 0x7f64540d1bc0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f6444001670 con 0x7f64540d1bc0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 >> v1:172.21.15.150:6789/0 conn(0x7f64540d1bc0 legacy=0x7f64540b3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643c0039b0 0x7f643c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f643c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f643c004d40 0x7f643c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64540cecb0 con 0x7f643c004d40 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f643c00b9c0 con 0x7f643c0039b0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f643c00bb40 con 0x7f64540ce4d0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.559+0000 7f6458b34640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f643c004d40 0x7f643c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f64537fe640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f643c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6453fff640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643c0039b0 0x7f643c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 -- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 msgr2=0x7f643c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64540ce4d0 0x7f643c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 -- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643c0039b0 msgr2=0x7f643c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643c0039b0 0x7f643c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 -- 172.21.15.150:0/2974572834 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f643c003520 con 0x7f643c004d40 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6453fff640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f643c0039b0 0x7f643c003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.560+0000 7f6458b34640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f643c004d40 0x7f643c005150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6454164fd0 tx=0x7f643c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113b3a884a131849 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c037070 con 0x7f643c004d40 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f644c032620 con 0x7f643c004d40 2024-04-25T14:02:29.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f6452ffd640 1 -- 172.21.15.150:0/2974572834 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c009c10 con 0x7f643c004d40 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 -- 172.21.15.150:0/2974572834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f643c004d40 msgr2=0x7f643c005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 --2- 172.21.15.150:0/2974572834 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f643c004d40 0x7f643c005150 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6454164fd0 tx=0x7f643c005830 comp rx=0 tx=0).stop 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f6458b34640 1 -- 172.21.15.150:0/2974572834 reap_dead start 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 -- 172.21.15.150:0/2974572834 shutdown_connections 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 -- 172.21.15.150:0/2974572834 >> 172.21.15.150:0/2974572834 conn(0x7f645409b4a0 msgr2=0x7f64540b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:29.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 -- 172.21.15.150:0/2974572834 shutdown_connections 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.561+0000 7f645a597640 1 -- 172.21.15.150:0/2974572834 wait complete. 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.563+0000 7f645a597640 1 Processor -- start 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.563+0000 7f645a597640 1 -- start start 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6454160570 0x7f6454161b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64541594e0 0x7f6454162830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f64540b3420 con 0x7f64541598f0 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f64540b32a0 con 0x7f6454160570 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f64540b35a0 con 0x7f64541594e0 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38174/0 (socket says 172.21.15.150:38174) 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 -- 172.21.15.150:0/3349858083 learned_addr learned my addr 172.21.15.150:0/3349858083 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6453fff640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6454160570 0x7f6454161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f64537fe640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64541594e0 0x7f6454162830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 -- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64541594e0 msgr2=0x7f6454162830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64541594e0 0x7f6454162830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 -- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6454160570 msgr2=0x7f6454161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6454160570 0x7f6454161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 -- 172.21.15.150:0/3349858083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6454160af0 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6458b34640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f644c02ae60 tx=0x7f644c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c037040 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f644c032420 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6454160dd0 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444007d50 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.565+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6454161390 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.566+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6454076a40 con 0x7f64541598f0 2024-04-25T14:02:29.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.567+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f644c045030 con 0x7f64541598f0 2024-04-25T14:02:29.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.568+0000 7f6450ff9640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 0x7f64400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:29.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.568+0000 7f6453fff640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 0x7f64400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:29.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.568+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f644c0b5460 con 0x7f64541598f0 2024-04-25T14:02:29.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.568+0000 7f6453fff640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 0x7f64400769c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6444000c00 tx=0x7f6444007970 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:29.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.572+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f644c04c020 con 0x7f64541598f0 2024-04-25T14:02:29.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:29.787+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f645409baf0 con 0x7f64541598f0 2024-04-25T14:02:30.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.235+0000 7f6450ff9640 1 -- 172.21.15.150:0/3349858083 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f644c0325f0 con 0x7f64541598f0 2024-04-25T14:02:30.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 msgr2=0x7f64400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 0x7f64400769c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6444000c00 tx=0x7f6444007970 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 msgr2=0x7f6454162d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f644c02ae60 tx=0x7f644c009750 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 shutdown_connections 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f64400744b0 0x7f64400769c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f64541598f0 0x7f6454162d70 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f64541594e0 0x7f6454162830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 --2- 172.21.15.150:0/3349858083 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6454160570 0x7f6454161b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 >> 172.21.15.150:0/3349858083 conn(0x7f645409b4a0 msgr2=0x7f645415f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 shutdown_connections 2024-04-25T14:02:30.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.238+0000 7f645a597640 1 -- 172.21.15.150:0/3349858083 wait complete. 2024-04-25T14:02:30.250 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-25T14:02:30.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T14:02:30.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.361+0000 7f33f9c6e640 1 Processor -- start 2024-04-25T14:02:30.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.362+0000 7f33f9c6e640 1 -- start start 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f9c6e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33f41600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f9c6e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33f40d1750 con 0x7f33f4160630 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f9c6e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33f40d55d0 con 0x7f33f40d1000 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f9c6e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f33f4160a40 con 0x7f33f40d3de0 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f2ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f33f4160630 0x7f33f40d08f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55450/0 (socket says 172.21.15.150:55450) 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f2ffd640 1 -- 172.21.15.150:0/2619305141 learned_addr learned my addr 172.21.15.150:0/2619305141 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 610446379 0 0) 0x7f33f40d1750 con 0x7f33f4160630 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33dc003520 con 0x7f33f4160630 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f37fe640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33f41600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.363+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1230518138 0 0) 0x7f33dc003520 con 0x7f33f4160630 2024-04-25T14:02:30.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 msgr2=0x7f33f41600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33f41600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 >> v1:172.21.15.187:6789/0 conn(0x7f33f40d1000 legacy=0x7f33f406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f40d4530 con 0x7f33f4160630 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f33e8001670 con 0x7f33f4160630 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 >> v1:172.21.15.150:6789/0 conn(0x7f33f4160630 legacy=0x7f33f40d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33dc0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33dc003f60 0x7f33dc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc004d40 0x7f33dc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33f4160a40 con 0x7f33dc003f60 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00bb50 con 0x7f33dc004d40 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f33dc00bd50 con 0x7f33f40d3de0 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33dc003f60 0x7f33dc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f37fe640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33dc0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f3fff640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc004d40 0x7f33dc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 -- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 msgr2=0x7f33dc0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f40d3de0 0x7f33dc0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 -- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc004d40 msgr2=0x7f33dc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc004d40 0x7f33dc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.364+0000 7f33f2ffd640 1 -- 172.21.15.150:0/2619305141 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33dc003520 con 0x7f33dc003f60 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.365+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33dc003f60 0x7f33dc004800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c240 tx=0x7f33dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d80b65744b963e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.365+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8007d10 con 0x7f33dc003f60 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.365+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e8005ad0 con 0x7f33dc003f60 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.365+0000 7f33f27fc640 1 -- 172.21.15.150:0/2619305141 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e8007070 con 0x7f33dc003f60 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 -- 172.21.15.150:0/2619305141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33dc003f60 msgr2=0x7f33dc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/2619305141 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33dc003f60 0x7f33dc004800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f33e800c240 tx=0x7f33dc005830 comp rx=0 tx=0).stop 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f3fff640 1 -- 172.21.15.150:0/2619305141 reap_dead start 2024-04-25T14:02:30.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 -- 172.21.15.150:0/2619305141 shutdown_connections 2024-04-25T14:02:30.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 -- 172.21.15.150:0/2619305141 >> 172.21.15.150:0/2619305141 conn(0x7f33f40d25c0 msgr2=0x7f33f40d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:30.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 -- 172.21.15.150:0/2619305141 shutdown_connections 2024-04-25T14:02:30.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.366+0000 7f33f9c6e640 1 -- 172.21.15.150:0/2619305141 wait complete. 2024-04-25T14:02:30.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.367+0000 7f33f9c6e640 1 Processor -- start 2024-04-25T14:02:30.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.368+0000 7f33f9c6e640 1 -- start start 2024-04-25T14:02:30.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 0x7f33f40cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 0x7f33f406cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f33f40d4dd0 con 0x7f33f406d050 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f33f40d4af0 con 0x7f33dc003f60 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f9c6e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f33f40d50b0 con 0x7f33f414fdc0 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38184/0 (socket says 172.21.15.150:38184) 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 -- 172.21.15.150:0/3561241085 learned_addr learned my addr 172.21.15.150:0/3561241085 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f37fe640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 0x7f33f40cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 0x7f33f406cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 -- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 msgr2=0x7f33f406cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 0x7f33f406cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 -- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 msgr2=0x7f33f40cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:30.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 0x7f33f40cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 -- 172.21.15.150:0/3561241085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f406d460 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f2ffd640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 0x7f33f406cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f37fe640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 0x7f33f40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.369+0000 7f33f3fff640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33f40ccb90 tx=0x7f33ec00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.370+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec003b80 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.370+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33ec018070 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.370+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f40ca0f0 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.370+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0050d0 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.370+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f40ca8d0 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.371+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33f4076a40 con 0x7f33f406d050 2024-04-25T14:02:30.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.371+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f33ec005270 con 0x7f33f406d050 2024-04-25T14:02:30.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.372+0000 7f33dbfff640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 0x7f33e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:30.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.372+0000 7f33f37fe640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 0x7f33e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:30.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.372+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f33ec095400 con 0x7f33f406d050 2024-04-25T14:02:30.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.373+0000 7f33f37fe640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 0x7f33e00769c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33f408d210 tx=0x7f33e4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:30.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.377+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f33f406d460 con 0x7f33f406d050 2024-04-25T14:02:30.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:30.596+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f33f40ca380 con 0x7f33f406d050 2024-04-25T14:02:31.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.237+0000 7f33dbfff640 1 -- 172.21.15.150:0/3561241085 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f33ec05f8e0 con 0x7f33f406d050 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 msgr2=0x7f33e00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 0x7f33e00769c0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f33f408d210 tx=0x7f33e4008040 comp rx=0 tx=0).stop 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 msgr2=0x7f33f415b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f33f40ccb90 tx=0x7f33ec00fd40 comp rx=0 tx=0).stop 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 shutdown_connections 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f33e00744b0 0x7f33e00769c0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f33f406d050 0x7f33f415b920 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f33f414fdc0 0x7f33f406cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 --2- 172.21.15.150:0/3561241085 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f33dc003f60 0x7f33f40cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.239+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 >> 172.21.15.150:0/3561241085 conn(0x7f33f40d25c0 msgr2=0x7f33f40b6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:31.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.240+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 shutdown_connections 2024-04-25T14:02:31.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.240+0000 7f33f9c6e640 1 -- 172.21.15.150:0/3561241085 wait complete. 2024-04-25T14:02:31.253 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T14:02:31.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:31.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.364+0000 7ff6a06bd640 1 Processor -- start 2024-04-25T14:02:31.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.364+0000 7ff6a06bd640 1 -- start start 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff6a06bd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff6981647a0 0x7ff6980cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff6a06bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 0x7ff6980b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff6a06bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff698164bb0 con 0x7ff698160070 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff6a06bd640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff6980b8380 con 0x7ff6981647a0 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff6a06bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6980d2c70 con 0x7ff6980b5b10 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff69dc58640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff698160070 0x7ff6980b5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55458/0 (socket says 172.21.15.150:55458) 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff69dc58640 1 -- 172.21.15.150:0/3940726335 learned_addr learned my addr 172.21.15.150:0/3940726335 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.365+0000 7ff69e459640 1 --2- 172.21.15.150:0/3940726335 >> v2:172.21.15.187:3300/0 conn(0x7ff6981647a0 0x7ff6980cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69ec5a640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 0x7ff6980b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566576892 0 0) 0x7ff698164bb0 con 0x7ff698160070 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff684003520 con 0x7ff698160070 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2802703329 0 0) 0x7ff684003520 con 0x7ff698160070 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69e459640 1 -- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 msgr2=0x7ff6980b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69e459640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 0x7ff6980b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69e459640 1 -- 172.21.15.150:0/3940726335 >> v1:172.21.15.150:6789/0 conn(0x7ff698160070 legacy=0x7ff6980b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69e459640 1 -- 172.21.15.150:0/3940726335 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6980d2f50 con 0x7ff6981647a0 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.366+0000 7ff69ec5a640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 0x7ff6980b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.367+0000 7ff69e459640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff6980cfcc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff694009d90 tx=0x7ff69402a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64f891d5627bd662 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.367+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff688001670 con 0x7ff6981647a0 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.367+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff694007540 con 0x7ff6981647a0 2024-04-25T14:02:31.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.367+0000 7ff69d457640 1 -- 172.21.15.150:0/3940726335 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6940078f0 con 0x7ff6981647a0 2024-04-25T14:02:31.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 -- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 msgr2=0x7ff6980cfcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff6980cfcc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff694009d90 tx=0x7ff69402a560 comp rx=0 tx=0).stop 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 -- 172.21.15.150:0/3940726335 shutdown_connections 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff6980b5b10 0x7ff6980b8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/3940726335 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff6980cfcc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 -- 172.21.15.150:0/3940726335 >> 172.21.15.150:0/3940726335 conn(0x7ff6980d0400 msgr2=0x7ff6980d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 -- 172.21.15.150:0/3940726335 shutdown_connections 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.368+0000 7ff6a06bd640 1 -- 172.21.15.150:0/3940726335 wait complete. 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.369+0000 7ff6a06bd640 1 Processor -- start 2024-04-25T14:02:31.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.369+0000 7ff6a06bd640 1 -- start start 2024-04-25T14:02:31.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff698160070 0x7ff698159500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff698159a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff6980cdfd0 con 0x7ff698157c70 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff6980b7fa0 con 0x7ff6981647a0 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff6a06bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff6980ce150 con 0x7ff698160070 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38198/0 (socket says 172.21.15.150:38198) 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 -- 172.21.15.150:0/637868755 learned_addr learned my addr 172.21.15.150:0/637868755 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69dc58640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff698159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69e459640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff698160070 0x7ff698159500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 -- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff698160070 msgr2=0x7ff698159500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff698160070 0x7ff698159500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 -- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 msgr2=0x7ff698159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff698159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 -- 172.21.15.150:0/637868755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff698160800 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff69ec5a640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff69000e880 tx=0x7ff69000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.371+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff690017070 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.372+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff690004020 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.372+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff698160c70 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.372+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6900043f0 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.372+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff698155a90 con 0x7ff698157c70 2024-04-25T14:02:31.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.372+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6981555c0 con 0x7ff698157c70 2024-04-25T14:02:31.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.373+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff69000c940 con 0x7ff698157c70 2024-04-25T14:02:31.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.374+0000 7ff68effd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 0x7ff67c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.374+0000 7ff69e459640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 0x7ff67c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.374+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff6900952e0 con 0x7ff698157c70 2024-04-25T14:02:31.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.374+0000 7ff69e459640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 0x7ff67c0769f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff694034000 tx=0x7ff694033040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.378+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff69005f850 con 0x7ff698157c70 2024-04-25T14:02:31.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.593+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6981609d0 con 0x7ff698157c70 2024-04-25T14:02:31.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.593+0000 7ff68effd640 1 -- 172.21.15.150:0/637868755 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7ff69005f150 con 0x7ff698157c70 2024-04-25T14:02:31.594 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:31.594 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:31.594 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 msgr2=0x7ff67c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 0x7ff67c0769f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff694034000 tx=0x7ff694033040 comp rx=0 tx=0).stop 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 msgr2=0x7ff698161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff69000e880 tx=0x7ff69000ed40 comp rx=0 tx=0).stop 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 shutdown_connections 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff67c0744e0 0x7ff67c0769f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff698157c70 0x7ff698161190 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff6981647a0 0x7ff698159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 --2- 172.21.15.150:0/637868755 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff698160070 0x7ff698159500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 >> 172.21.15.150:0/637868755 conn(0x7ff6980d0400 msgr2=0x7ff69806df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:31.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 shutdown_connections 2024-04-25T14:02:31.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.596+0000 7ff6a06bd640 1 -- 172.21.15.150:0/637868755 wait complete. 2024-04-25T14:02:31.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:31.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.713+0000 7fb74aebd640 1 Processor -- start 2024-04-25T14:02:31.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.713+0000 7fb74aebd640 1 -- start start 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74aebd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 0x7fb744160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74aebd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb744164bc0 con 0x7fb7440d2e90 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74aebd640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7440ce680 con 0x7fb7440b2b10 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74aebd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7440b7db0 con 0x7fb7440b5ab0 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb743fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb7440d2e90 0x7fb7440b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55462/0 (socket says 172.21.15.150:55462) 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb743fff640 1 -- 172.21.15.150:0/923256082 learned_addr learned my addr 172.21.15.150:0/923256082 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74945a640 1 --2- 172.21.15.150:0/923256082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 0x7fb744160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 759862299 0 0) 0x7fb744164bc0 con 0x7fb7440d2e90 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb72c003520 con 0x7fb7440d2e90 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2650879536 0 0) 0x7fb7440ce680 con 0x7fb7440b2b10 2024-04-25T14:02:31.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb744164bc0 con 0x7fb7440b2b10 2024-04-25T14:02:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74945a640 1 -- 172.21.15.150:0/923256082 >> v1:172.21.15.187:6789/0 conn(0x7fb7440b2b10 legacy=0x7fb7440cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74945a640 1 -- 172.21.15.150:0/923256082 >> v1:172.21.15.150:6789/0 conn(0x7fb7440d2e90 legacy=0x7fb7440b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74945a640 1 -- 172.21.15.150:0/923256082 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7441648e0 con 0x7fb7440b5ab0 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb74945a640 1 --2- 172.21.15.150:0/923256082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 0x7fb744160220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009dc0 tx=0x7fb73c02a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=778b2316c12eac84 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c007740 con 0x7fb7440b5ab0 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb73c0078e0 con 0x7fb7440b5ab0 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.716+0000 7fb7437fe640 1 -- 172.21.15.150:0/923256082 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c007c90 con 0x7fb7440b5ab0 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 -- 172.21.15.150:0/923256082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 msgr2=0x7fb744160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 --2- 172.21.15.150:0/923256082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 0x7fb744160220 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009dc0 tx=0x7fb73c02a5b0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 -- 172.21.15.150:0/923256082 shutdown_connections 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 --2- 172.21.15.150:0/923256082 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb7440b5ab0 0x7fb744160220 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 -- 172.21.15.150:0/923256082 >> 172.21.15.150:0/923256082 conn(0x7fb7440d03a0 msgr2=0x7fb7440d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 -- 172.21.15.150:0/923256082 shutdown_connections 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.717+0000 7fb74aebd640 1 -- 172.21.15.150:0/923256082 wait complete. 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.719+0000 7fb74aebd640 1 Processor -- start 2024-04-25T14:02:31.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.719+0000 7fb74aebd640 1 -- start start 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7440d2e90 0x7fb7441512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb744143aa0 0x7fb7440d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb7440b8350 con 0x7fb7440d1a30 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb744164690 con 0x7fb7440d2e90 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74aebd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb7440b9720 con 0x7fb744143aa0 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74945a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.720+0000 7fb74945a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38214/0 (socket says 172.21.15.150:38214) 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 -- 172.21.15.150:0/2503082681 learned_addr learned my addr 172.21.15.150:0/2503082681 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:31.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb743fff640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb744143aa0 0x7fb7440d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb748c59640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7440d2e90 0x7fb7441512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 -- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb744143aa0 msgr2=0x7fb7440d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb744143aa0 0x7fb7440d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 -- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7440d2e90 msgr2=0x7fb7441512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7440d2e90 0x7fb7441512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 -- 172.21.15.150:0/2503082681 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7440d1e40 con 0x7fb7440d1a30 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74945a640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009eb0 tx=0x7fb73c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c03f070 con 0x7fb7440d1a30 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb73c031c70 con 0x7fb7440d1a30 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73c009450 con 0x7fb7440d1a30 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb744070290 con 0x7fb7440d1a30 2024-04-25T14:02:31.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.721+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb744070850 con 0x7fb7440d1a30 2024-04-25T14:02:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.722+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb744076a40 con 0x7fb7440d1a30 2024-04-25T14:02:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.723+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb73c03b410 con 0x7fb7440d1a30 2024-04-25T14:02:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.723+0000 7fb7417fa640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 0x7fb7300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.724+0000 7fb748c59640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 0x7fb7300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:31.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.724+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb73c0b6b30 con 0x7fb7440d1a30 2024-04-25T14:02:31.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.724+0000 7fb748c59640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 0x7fb7300769c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb734007fd0 tx=0x7fb734009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:31.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.729+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb7440d1e40 con 0x7fb7440d1a30 2024-04-25T14:02:31.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.904+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7440b3060 con 0x7fb7440d1a30 2024-04-25T14:02:31.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.905+0000 7fb7417fa640 1 -- 172.21.15.150:0/2503082681 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7fb73c031e20 con 0x7fb7440d1a30 2024-04-25T14:02:31.905 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:31.906 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:31.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 msgr2=0x7fb7300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 0x7fb7300769c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb734007fd0 tx=0x7fb734009040 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 msgr2=0x7fb744152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb73c009eb0 tx=0x7fb73c004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 shutdown_connections 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb7300744b0 0x7fb7300769c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb7440d1a30 0x7fb744152a30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb744143aa0 0x7fb7440d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 --2- 172.21.15.150:0/2503082681 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb7440d2e90 0x7fb7441512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 >> 172.21.15.150:0/2503082681 conn(0x7fb7440d03a0 msgr2=0x7fb7440b4a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 shutdown_connections 2024-04-25T14:02:31.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:31.907+0000 7fb74aebd640 1 -- 172.21.15.150:0/2503082681 wait complete. 2024-04-25T14:02:31.919 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T14:02:31.919 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:02:32.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 Processor -- start 2024-04-25T14:02:32.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 -- start start 2024-04-25T14:02:32.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb140b4f60 0x7fbb140cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbb14161160 0x7fbb1415cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb140b3f00 con 0x7fbb140cfe30 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbb140b4ad0 con 0x7fbb140b4f60 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb19fba640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbb140b41a0 con 0x7fbb14161160 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb137fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb140b4f60 0x7fbb140cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb12ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbb140cfe30 0x7fbb14160a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55478/0 (socket says 172.21.15.150:55478) 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb137fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbb140b4f60 0x7fbb140cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38222/0 (socket says 172.21.15.150:38222) 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb12ffd640 1 -- 172.21.15.150:0/2812520737 learned_addr learned my addr 172.21.15.150:0/2812520737 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.027+0000 7fbb13fff640 1 --2- 172.21.15.150:0/2812520737 >> v2:172.21.15.187:3300/0 conn(0x7fbb14161160 0x7fbb1415cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb137fe640 1 -- 172.21.15.150:0/2812520737 >> v2:172.21.15.187:3300/0 conn(0x7fbb14161160 msgr2=0x7fbb1415cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb137fe640 1 --2- 172.21.15.150:0/2812520737 >> v2:172.21.15.187:3300/0 conn(0x7fbb14161160 0x7fbb1415cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb137fe640 1 -- 172.21.15.150:0/2812520737 >> v1:172.21.15.150:6789/0 conn(0x7fbb140cfe30 legacy=0x7fbb14160a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb127fc640 1 -- 172.21.15.150:0/2812520737 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495312113 0 0) 0x7fbb140b3f00 con 0x7fbb140cfe30 2024-04-25T14:02:32.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb137fe640 1 -- 172.21.15.150:0/2812520737 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb140b4480 con 0x7fbb140b4f60 2024-04-25T14:02:32.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb137fe640 1 --2- 172.21.15.150:0/2812520737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb140b4f60 0x7fbb140cf8f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb140b8880 tx=0x7fbb0402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f0e0d8c6aff9417 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb127fc640 1 -- 172.21.15.150:0/2812520737 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04037070 con 0x7fbb140b4f60 2024-04-25T14:02:32.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb127fc640 1 -- 172.21.15.150:0/2812520737 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb04032440 con 0x7fbb140b4f60 2024-04-25T14:02:32.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.028+0000 7fbb127fc640 1 -- 172.21.15.150:0/2812520737 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04032810 con 0x7fbb140b4f60 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 -- 172.21.15.150:0/2812520737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb140b4f60 msgr2=0x7fbb140cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2812520737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb140b4f60 0x7fbb140cf8f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fbb140b8880 tx=0x7fbb0402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 -- 172.21.15.150:0/2812520737 shutdown_connections 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2812520737 >> v2:172.21.15.187:3300/0 conn(0x7fbb14161160 0x7fbb1415cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2812520737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb140b4f60 0x7fbb140cf8f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 -- 172.21.15.150:0/2812520737 >> 172.21.15.150:0/2812520737 conn(0x7fbb1409b4a0 msgr2=0x7fbb1409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 -- 172.21.15.150:0/2812520737 shutdown_connections 2024-04-25T14:02:32.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.029+0000 7fbb19fba640 1 -- 172.21.15.150:0/2812520737 wait complete. 2024-04-25T14:02:32.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.031+0000 7fbb19fba640 1 Processor -- start 2024-04-25T14:02:32.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.031+0000 7fbb19fba640 1 -- start start 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.032+0000 7fbb19fba640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb140b4f60 0x7fbb14157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.032+0000 7fbb19fba640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb140cfe30 0x7fbb140cc020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.032+0000 7fbb19fba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb19fba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbb141618b0 con 0x7fbb1406ce00 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb19fba640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbb14161730 con 0x7fbb140cfe30 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb19fba640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbb1415d520 con 0x7fbb140b4f60 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38230/0 (socket says 172.21.15.150:38230) 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 -- 172.21.15.150:0/2469362820 learned_addr learned my addr 172.21.15.150:0/2469362820 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb137fe640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb140b4f60 0x7fbb14157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb12ffd640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb140cfe30 0x7fbb140cc020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 -- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb140b4f60 msgr2=0x7fbb14157e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb140b4f60 0x7fbb14157e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 -- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb140cfe30 msgr2=0x7fbb140cc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb140cfe30 0x7fbb140cc020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 -- 172.21.15.150:0/2469362820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb1406d750 con 0x7fbb1406ce00 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb13fff640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00ef90 tx=0x7fbb0c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c018020 con 0x7fbb1406ce00 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb0c014480 con 0x7fbb1406ce00 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb1415c5e0 con 0x7fbb1406ce00 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c009e60 con 0x7fbb1406ce00 2024-04-25T14:02:32.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.033+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb141f6c20 con 0x7fbb1406ce00 2024-04-25T14:02:32.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.034+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb14076a40 con 0x7fbb1406ce00 2024-04-25T14:02:32.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.035+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbb0c005d10 con 0x7fbb1406ce00 2024-04-25T14:02:32.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.036+0000 7fbafbfff640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 0x7fbb000769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.036+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fbb0c0960b0 con 0x7fbb1406ce00 2024-04-25T14:02:32.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.036+0000 7fbb137fe640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 0x7fbb000769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.036+0000 7fbb137fe640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 0x7fbb000769c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb04034000 tx=0x7fbb0402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.041+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb0c022030 con 0x7fbb1406ce00 2024-04-25T14:02:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.215+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbb140cfa80 con 0x7fbb000744b0 2024-04-25T14:02:32.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.224+0000 7fbafbfff640 1 -- 172.21.15.150:0/2469362820 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7fbb140cfa80 con 0x7fbb000744b0 2024-04-25T14:02:32.228 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:32.230 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:02:32.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 msgr2=0x7fbb000769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 0x7fbb000769c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbb04034000 tx=0x7fbb0402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:32.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 msgr2=0x7fbb1406d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00ef90 tx=0x7fbb0c00c490 comp rx=0 tx=0).stop 2024-04-25T14:02:32.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 shutdown_connections 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbb000744b0 0x7fbb000769c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbb1406ce00 0x7fbb1406d210 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbb140cfe30 0x7fbb140cc020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 --2- 172.21.15.150:0/2469362820 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbb140b4f60 0x7fbb14157e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 >> 172.21.15.150:0/2469362820 conn(0x7fbb1409b4a0 msgr2=0x7fbb14070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 shutdown_connections 2024-04-25T14:02:32.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.227+0000 7fbb19fba640 1 -- 172.21.15.150:0/2469362820 wait complete. 2024-04-25T14:02:32.239 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-25T14:02:30.591173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003104"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180578+0000","last_change":"2024-04-25T14:02:08.147737+0000","last_active":"2024-04-25T14:02:14.180578+0000","last_peered":"2024-04-25T14:02:14.180578+0000","last_clean":"2024-04-25T14:02:14.180578+0000","last_became_active":"2024-04-25T14:02:08.147072+0000","last_became_peered":"2024-04-25T14:02:08.147072+0000","last_unstale":"2024-04-25T14:02:14.180578+0000","last_undegraded":"2024-04-25T14:02:14.180578+0000","last_fullsized":"2024-04-25T14:02:14.180578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621342+0000","last_change":"2024-04-25T14:02:09.168014+0000","last_active":"2024-04-25T14:02:14.621342+0000","last_peered":"2024-04-25T14:02:14.621342+0000","last_clean":"2024-04-25T14:02:14.621342+0000","last_became_active":"2024-04-25T14:02:09.166846+0000","last_became_peered":"2024-04-25T14:02:09.166846+0000","last_unstale":"2024-04-25T14:02:14.621342+0000","last_undegraded":"2024-04-25T14:02:14.621342+0000","last_fullsized":"2024-04-25T14:02:14.621342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619101+0000","last_change":"2024-04-25T14:02:13.203189+0000","last_active":"2024-04-25T14:02:14.619101+0000","last_peered":"2024-04-25T14:02:14.619101+0000","last_clean":"2024-04-25T14:02:14.619101+0000","last_became_active":"2024-04-25T14:02:13.201743+0000","last_became_peered":"2024-04-25T14:02:13.201743+0000","last_unstale":"2024-04-25T14:02:14.619101+0000","last_undegraded":"2024-04-25T14:02:14.619101+0000","last_fullsized":"2024-04-25T14:02:14.619101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621459+0000","last_change":"2024-04-25T14:02:08.147961+0000","last_active":"2024-04-25T14:02:14.621459+0000","last_peered":"2024-04-25T14:02:14.621459+0000","last_clean":"2024-04-25T14:02:14.621459+0000","last_became_active":"2024-04-25T14:02:08.147652+0000","last_became_peered":"2024-04-25T14:02:08.147652+0000","last_unstale":"2024-04-25T14:02:14.621459+0000","last_undegraded":"2024-04-25T14:02:14.621459+0000","last_fullsized":"2024-04-25T14:02:14.621459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180307+0000","last_change":"2024-04-25T14:02:09.173887+0000","last_active":"2024-04-25T14:02:14.180307+0000","last_peered":"2024-04-25T14:02:14.180307+0000","last_clean":"2024-04-25T14:02:14.180307+0000","last_became_active":"2024-04-25T14:02:09.173583+0000","last_became_peered":"2024-04-25T14:02:09.173583+0000","last_unstale":"2024-04-25T14:02:14.180307+0000","last_undegraded":"2024-04-25T14:02:14.180307+0000","last_fullsized":"2024-04-25T14:02:14.180307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619293+0000","last_change":"2024-04-25T14:02:13.202095+0000","last_active":"2024-04-25T14:02:14.619293+0000","last_peered":"2024-04-25T14:02:14.619293+0000","last_clean":"2024-04-25T14:02:14.619293+0000","last_became_active":"2024-04-25T14:02:13.201620+0000","last_became_peered":"2024-04-25T14:02:13.201620+0000","last_unstale":"2024-04-25T14:02:14.619293+0000","last_undegraded":"2024-04-25T14:02:14.619293+0000","last_fullsized":"2024-04-25T14:02:14.619293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:25.777824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181439+0000","last_change":"2024-04-25T14:02:08.139279+0000","last_active":"2024-04-25T14:02:14.181439+0000","last_peered":"2024-04-25T14:02:14.181439+0000","last_clean":"2024-04-25T14:02:14.181439+0000","last_became_active":"2024-04-25T14:02:08.138923+0000","last_became_peered":"2024-04-25T14:02:08.138923+0000","last_unstale":"2024-04-25T14:02:14.181439+0000","last_undegraded":"2024-04-25T14:02:14.181439+0000","last_fullsized":"2024-04-25T14:02:14.181439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:22.987870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619690+0000","last_change":"2024-04-25T14:02:09.162551+0000","last_active":"2024-04-25T14:02:14.619690+0000","last_peered":"2024-04-25T14:02:14.619690+0000","last_clean":"2024-04-25T14:02:14.619690+0000","last_became_active":"2024-04-25T14:02:09.161794+0000","last_became_peered":"2024-04-25T14:02:09.161794+0000","last_unstale":"2024-04-25T14:02:14.619690+0000","last_undegraded":"2024-04-25T14:02:14.619690+0000","last_fullsized":"2024-04-25T14:02:14.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619746+0000","last_change":"2024-04-25T14:02:13.203879+0000","last_active":"2024-04-25T14:02:14.619746+0000","last_peered":"2024-04-25T14:02:14.619746+0000","last_clean":"2024-04-25T14:02:14.619746+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619746+0000","last_undegraded":"2024-04-25T14:02:14.619746+0000","last_fullsized":"2024-04-25T14:02:14.619746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.794247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180468+0000","last_change":"2024-04-25T14:02:08.148407+0000","last_active":"2024-04-25T14:02:14.180468+0000","last_peered":"2024-04-25T14:02:14.180468+0000","last_clean":"2024-04-25T14:02:14.180468+0000","last_became_active":"2024-04-25T14:02:08.148049+0000","last_became_peered":"2024-04-25T14:02:08.148049+0000","last_unstale":"2024-04-25T14:02:14.180468+0000","last_undegraded":"2024-04-25T14:02:14.180468+0000","last_fullsized":"2024-04-25T14:02:14.180468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620378+0000","last_change":"2024-04-25T14:02:09.162054+0000","last_active":"2024-04-25T14:02:14.620378+0000","last_peered":"2024-04-25T14:02:14.620378+0000","last_clean":"2024-04-25T14:02:14.620378+0000","last_became_active":"2024-04-25T14:02:09.161561+0000","last_became_peered":"2024-04-25T14:02:09.161561+0000","last_unstale":"2024-04-25T14:02:14.620378+0000","last_undegraded":"2024-04-25T14:02:14.620378+0000","last_fullsized":"2024-04-25T14:02:14.620378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620224+0000","last_change":"2024-04-25T14:02:13.202231+0000","last_active":"2024-04-25T14:02:14.620224+0000","last_peered":"2024-04-25T14:02:14.620224+0000","last_clean":"2024-04-25T14:02:14.620224+0000","last_became_active":"2024-04-25T14:02:13.201805+0000","last_became_peered":"2024-04-25T14:02:13.201805+0000","last_unstale":"2024-04-25T14:02:14.620224+0000","last_undegraded":"2024-04-25T14:02:14.620224+0000","last_fullsized":"2024-04-25T14:02:14.620224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:12.379500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181919+0000","last_change":"2024-04-25T14:02:09.164630+0000","last_active":"2024-04-25T14:02:14.181919+0000","last_peered":"2024-04-25T14:02:14.181919+0000","last_clean":"2024-04-25T14:02:14.181919+0000","last_became_active":"2024-04-25T14:02:09.164292+0000","last_became_peered":"2024-04-25T14:02:09.164292+0000","last_unstale":"2024-04-25T14:02:14.181919+0000","last_undegraded":"2024-04-25T14:02:14.181919+0000","last_fullsized":"2024-04-25T14:02:14.181919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180969+0000","last_change":"2024-04-25T14:02:08.136088+0000","last_active":"2024-04-25T14:02:14.180969+0000","last_peered":"2024-04-25T14:02:14.180969+0000","last_clean":"2024-04-25T14:02:14.180969+0000","last_became_active":"2024-04-25T14:02:08.135716+0000","last_became_peered":"2024-04-25T14:02:08.135716+0000","last_unstale":"2024-04-25T14:02:14.180969+0000","last_undegraded":"2024-04-25T14:02:14.180969+0000","last_fullsized":"2024-04-25T14:02:14.180969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:27.040903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620573+0000","last_change":"2024-04-25T14:02:13.196592+0000","last_active":"2024-04-25T14:02:14.620573+0000","last_peered":"2024-04-25T14:02:14.620573+0000","last_clean":"2024-04-25T14:02:14.620573+0000","last_became_active":"2024-04-25T14:02:13.196120+0000","last_became_peered":"2024-04-25T14:02:13.196120+0000","last_unstale":"2024-04-25T14:02:14.620573+0000","last_undegraded":"2024-04-25T14:02:14.620573+0000","last_fullsized":"2024-04-25T14:02:14.620573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620568+0000","last_change":"2024-04-25T14:02:09.163625+0000","last_active":"2024-04-25T14:02:14.620568+0000","last_peered":"2024-04-25T14:02:14.620568+0000","last_clean":"2024-04-25T14:02:14.620568+0000","last_became_active":"2024-04-25T14:02:09.163320+0000","last_became_peered":"2024-04-25T14:02:09.163320+0000","last_unstale":"2024-04-25T14:02:14.620568+0000","last_undegraded":"2024-04-25T14:02:14.620568+0000","last_fullsized":"2024-04-25T14:02:14.620568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619947+0000","last_change":"2024-04-25T14:02:08.140552+0000","last_active":"2024-04-25T14:02:14.619947+0000","last_peered":"2024-04-25T14:02:14.619947+0000","last_clean":"2024-04-25T14:02:14.619947+0000","last_became_active":"2024-04-25T14:02:08.139997+0000","last_became_peered":"2024-04-25T14:02:08.139997+0000","last_unstale":"2024-04-25T14:02:14.619947+0000","last_undegraded":"2024-04-25T14:02:14.619947+0000","last_fullsized":"2024-04-25T14:02:14.619947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620055+0000","last_change":"2024-04-25T14:02:13.204879+0000","last_active":"2024-04-25T14:02:14.620055+0000","last_peered":"2024-04-25T14:02:14.620055+0000","last_clean":"2024-04-25T14:02:14.620055+0000","last_became_active":"2024-04-25T14:02:13.203934+0000","last_became_peered":"2024-04-25T14:02:13.203934+0000","last_unstale":"2024-04-25T14:02:14.620055+0000","last_undegraded":"2024-04-25T14:02:14.620055+0000","last_fullsized":"2024-04-25T14:02:14.620055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619890+0000","last_change":"2024-04-25T14:02:09.157409+0000","last_active":"2024-04-25T14:02:14.619890+0000","last_peered":"2024-04-25T14:02:14.619890+0000","last_clean":"2024-04-25T14:02:14.619890+0000","last_became_active":"2024-04-25T14:02:09.157102+0000","last_became_peered":"2024-04-25T14:02:09.157102+0000","last_unstale":"2024-04-25T14:02:14.619890+0000","last_undegraded":"2024-04-25T14:02:14.619890+0000","last_fullsized":"2024-04-25T14:02:14.619890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619463+0000","last_change":"2024-04-25T14:02:08.138257+0000","last_active":"2024-04-25T14:02:14.619463+0000","last_peered":"2024-04-25T14:02:14.619463+0000","last_clean":"2024-04-25T14:02:14.619463+0000","last_became_active":"2024-04-25T14:02:08.137925+0000","last_became_peered":"2024-04-25T14:02:08.137925+0000","last_unstale":"2024-04-25T14:02:14.619463+0000","last_undegraded":"2024-04-25T14:02:14.619463+0000","last_fullsized":"2024-04-25T14:02:14.619463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619999+0000","last_change":"2024-04-25T14:02:13.204203+0000","last_active":"2024-04-25T14:02:14.619999+0000","last_peered":"2024-04-25T14:02:14.619999+0000","last_clean":"2024-04-25T14:02:14.619999+0000","last_became_active":"2024-04-25T14:02:13.203578+0000","last_became_peered":"2024-04-25T14:02:13.203578+0000","last_unstale":"2024-04-25T14:02:14.619999+0000","last_undegraded":"2024-04-25T14:02:14.619999+0000","last_fullsized":"2024-04-25T14:02:14.619999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620454+0000","last_change":"2024-04-25T14:02:09.153035+0000","last_active":"2024-04-25T14:02:14.620454+0000","last_peered":"2024-04-25T14:02:14.620454+0000","last_clean":"2024-04-25T14:02:14.620454+0000","last_became_active":"2024-04-25T14:02:09.152071+0000","last_became_peered":"2024-04-25T14:02:09.152071+0000","last_unstale":"2024-04-25T14:02:14.620454+0000","last_undegraded":"2024-04-25T14:02:14.620454+0000","last_fullsized":"2024-04-25T14:02:14.620454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:11.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180075+0000","last_change":"2024-04-25T14:02:08.141740+0000","last_active":"2024-04-25T14:02:14.180075+0000","last_peered":"2024-04-25T14:02:14.180075+0000","last_clean":"2024-04-25T14:02:14.180075+0000","last_became_active":"2024-04-25T14:02:08.141400+0000","last_became_peered":"2024-04-25T14:02:08.141400+0000","last_unstale":"2024-04-25T14:02:14.180075+0000","last_undegraded":"2024-04-25T14:02:14.180075+0000","last_fullsized":"2024-04-25T14:02:14.180075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619288+0000","last_change":"2024-04-25T14:02:13.202697+0000","last_active":"2024-04-25T14:02:14.619288+0000","last_peered":"2024-04-25T14:02:14.619288+0000","last_clean":"2024-04-25T14:02:14.619288+0000","last_became_active":"2024-04-25T14:02:13.201555+0000","last_became_peered":"2024-04-25T14:02:13.201555+0000","last_unstale":"2024-04-25T14:02:14.619288+0000","last_undegraded":"2024-04-25T14:02:14.619288+0000","last_fullsized":"2024-04-25T14:02:14.619288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:43.769087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180683+0000","last_change":"2024-04-25T14:02:09.166496+0000","last_active":"2024-04-25T14:02:14.180683+0000","last_peered":"2024-04-25T14:02:14.180683+0000","last_clean":"2024-04-25T14:02:14.180683+0000","last_became_active":"2024-04-25T14:02:09.166180+0000","last_became_peered":"2024-04-25T14:02:09.166180+0000","last_unstale":"2024-04-25T14:02:14.180683+0000","last_undegraded":"2024-04-25T14:02:14.180683+0000","last_fullsized":"2024-04-25T14:02:14.180683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181386+0000","last_change":"2024-04-25T14:02:08.143988+0000","last_active":"2024-04-25T14:02:14.181386+0000","last_peered":"2024-04-25T14:02:14.181386+0000","last_clean":"2024-04-25T14:02:14.181386+0000","last_became_active":"2024-04-25T14:02:08.143639+0000","last_became_peered":"2024-04-25T14:02:08.143639+0000","last_unstale":"2024-04-25T14:02:14.181386+0000","last_undegraded":"2024-04-25T14:02:14.181386+0000","last_fullsized":"2024-04-25T14:02:14.181386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620239+0000","last_change":"2024-04-25T14:02:13.193130+0000","last_active":"2024-04-25T14:02:14.620239+0000","last_peered":"2024-04-25T14:02:14.620239+0000","last_clean":"2024-04-25T14:02:14.620239+0000","last_became_active":"2024-04-25T14:02:13.192664+0000","last_became_peered":"2024-04-25T14:02:13.192664+0000","last_unstale":"2024-04-25T14:02:14.620239+0000","last_undegraded":"2024-04-25T14:02:14.620239+0000","last_fullsized":"2024-04-25T14:02:14.620239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621118+0000","last_change":"2024-04-25T14:02:09.175890+0000","last_active":"2024-04-25T14:02:14.621118+0000","last_peered":"2024-04-25T14:02:14.621118+0000","last_clean":"2024-04-25T14:02:14.621118+0000","last_became_active":"2024-04-25T14:02:09.175586+0000","last_became_peered":"2024-04-25T14:02:09.175586+0000","last_unstale":"2024-04-25T14:02:14.621118+0000","last_undegraded":"2024-04-25T14:02:14.621118+0000","last_fullsized":"2024-04-25T14:02:14.621118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619271+0000","last_change":"2024-04-25T14:02:08.141597+0000","last_active":"2024-04-25T14:02:14.619271+0000","last_peered":"2024-04-25T14:02:14.619271+0000","last_clean":"2024-04-25T14:02:14.619271+0000","last_became_active":"2024-04-25T14:02:08.141271+0000","last_became_peered":"2024-04-25T14:02:08.141271+0000","last_unstale":"2024-04-25T14:02:14.619271+0000","last_undegraded":"2024-04-25T14:02:14.619271+0000","last_fullsized":"2024-04-25T14:02:14.619271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619661+0000","last_change":"2024-04-25T14:02:13.208359+0000","last_active":"2024-04-25T14:02:14.619661+0000","last_peered":"2024-04-25T14:02:14.619661+0000","last_clean":"2024-04-25T14:02:14.619661+0000","last_became_active":"2024-04-25T14:02:13.204049+0000","last_became_peered":"2024-04-25T14:02:13.204049+0000","last_unstale":"2024-04-25T14:02:14.619661+0000","last_undegraded":"2024-04-25T14:02:14.619661+0000","last_fullsized":"2024-04-25T14:02:14.619661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620199+0000","last_change":"2024-04-25T14:02:09.153508+0000","last_active":"2024-04-25T14:02:14.620199+0000","last_peered":"2024-04-25T14:02:14.620199+0000","last_clean":"2024-04-25T14:02:14.620199+0000","last_became_active":"2024-04-25T14:02:09.153087+0000","last_became_peered":"2024-04-25T14:02:09.153087+0000","last_unstale":"2024-04-25T14:02:14.620199+0000","last_undegraded":"2024-04-25T14:02:14.620199+0000","last_fullsized":"2024-04-25T14:02:14.620199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181562+0000","last_change":"2024-04-25T14:02:08.149363+0000","last_active":"2024-04-25T14:02:14.181562+0000","last_peered":"2024-04-25T14:02:14.181562+0000","last_clean":"2024-04-25T14:02:14.181562+0000","last_became_active":"2024-04-25T14:02:08.148936+0000","last_became_peered":"2024-04-25T14:02:08.148936+0000","last_unstale":"2024-04-25T14:02:14.181562+0000","last_undegraded":"2024-04-25T14:02:14.181562+0000","last_fullsized":"2024-04-25T14:02:14.181562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:33.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180089+0000","last_change":"2024-04-25T14:02:13.201594+0000","last_active":"2024-04-25T14:02:14.180089+0000","last_peered":"2024-04-25T14:02:14.180089+0000","last_clean":"2024-04-25T14:02:14.180089+0000","last_became_active":"2024-04-25T14:02:13.201156+0000","last_became_peered":"2024-04-25T14:02:13.201156+0000","last_unstale":"2024-04-25T14:02:14.180089+0000","last_undegraded":"2024-04-25T14:02:14.180089+0000","last_fullsized":"2024-04-25T14:02:14.180089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620504+0000","last_change":"2024-04-25T14:02:09.164648+0000","last_active":"2024-04-25T14:02:14.620504+0000","last_peered":"2024-04-25T14:02:14.620504+0000","last_clean":"2024-04-25T14:02:14.620504+0000","last_became_active":"2024-04-25T14:02:09.164319+0000","last_became_peered":"2024-04-25T14:02:09.164319+0000","last_unstale":"2024-04-25T14:02:14.620504+0000","last_undegraded":"2024-04-25T14:02:14.620504+0000","last_fullsized":"2024-04-25T14:02:14.620504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620863+0000","last_change":"2024-04-25T14:02:08.148046+0000","last_active":"2024-04-25T14:02:14.620863+0000","last_peered":"2024-04-25T14:02:14.620863+0000","last_clean":"2024-04-25T14:02:14.620863+0000","last_became_active":"2024-04-25T14:02:08.147707+0000","last_became_peered":"2024-04-25T14:02:08.147707+0000","last_unstale":"2024-04-25T14:02:14.620863+0000","last_undegraded":"2024-04-25T14:02:14.620863+0000","last_fullsized":"2024-04-25T14:02:14.620863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619751+0000","last_change":"2024-04-25T14:02:13.197572+0000","last_active":"2024-04-25T14:02:14.619751+0000","last_peered":"2024-04-25T14:02:14.619751+0000","last_clean":"2024-04-25T14:02:14.619751+0000","last_became_active":"2024-04-25T14:02:13.197138+0000","last_became_peered":"2024-04-25T14:02:13.197138+0000","last_unstale":"2024-04-25T14:02:14.619751+0000","last_undegraded":"2024-04-25T14:02:14.619751+0000","last_fullsized":"2024-04-25T14:02:14.619751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619861+0000","last_change":"2024-04-25T14:02:09.163807+0000","last_active":"2024-04-25T14:02:14.619861+0000","last_peered":"2024-04-25T14:02:14.619861+0000","last_clean":"2024-04-25T14:02:14.619861+0000","last_became_active":"2024-04-25T14:02:09.163510+0000","last_became_peered":"2024-04-25T14:02:09.163510+0000","last_unstale":"2024-04-25T14:02:14.619861+0000","last_undegraded":"2024-04-25T14:02:14.619861+0000","last_fullsized":"2024-04-25T14:02:14.619861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620015+0000","last_change":"2024-04-25T14:02:08.148684+0000","last_active":"2024-04-25T14:02:14.620015+0000","last_peered":"2024-04-25T14:02:14.620015+0000","last_clean":"2024-04-25T14:02:14.620015+0000","last_became_active":"2024-04-25T14:02:08.148384+0000","last_became_peered":"2024-04-25T14:02:08.148384+0000","last_unstale":"2024-04-25T14:02:14.620015+0000","last_undegraded":"2024-04-25T14:02:14.620015+0000","last_fullsized":"2024-04-25T14:02:14.620015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620837+0000","last_change":"2024-04-25T14:02:13.204849+0000","last_active":"2024-04-25T14:02:14.620837+0000","last_peered":"2024-04-25T14:02:14.620837+0000","last_clean":"2024-04-25T14:02:14.620837+0000","last_became_active":"2024-04-25T14:02:13.204459+0000","last_became_peered":"2024-04-25T14:02:13.204459+0000","last_unstale":"2024-04-25T14:02:14.620837+0000","last_undegraded":"2024-04-25T14:02:14.620837+0000","last_fullsized":"2024-04-25T14:02:14.620837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619156+0000","last_change":"2024-04-25T14:02:09.153461+0000","last_active":"2024-04-25T14:02:14.619156+0000","last_peered":"2024-04-25T14:02:14.619156+0000","last_clean":"2024-04-25T14:02:14.619156+0000","last_became_active":"2024-04-25T14:02:09.153122+0000","last_became_peered":"2024-04-25T14:02:09.153122+0000","last_unstale":"2024-04-25T14:02:14.619156+0000","last_undegraded":"2024-04-25T14:02:14.619156+0000","last_fullsized":"2024-04-25T14:02:14.619156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619253+0000","last_change":"2024-04-25T14:02:08.141488+0000","last_active":"2024-04-25T14:02:14.619253+0000","last_peered":"2024-04-25T14:02:14.619253+0000","last_clean":"2024-04-25T14:02:14.619253+0000","last_became_active":"2024-04-25T14:02:08.141195+0000","last_became_peered":"2024-04-25T14:02:08.141195+0000","last_unstale":"2024-04-25T14:02:14.619253+0000","last_undegraded":"2024-04-25T14:02:14.619253+0000","last_fullsized":"2024-04-25T14:02:14.619253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620173+0000","last_change":"2024-04-25T14:02:13.202926+0000","last_active":"2024-04-25T14:02:14.620173+0000","last_peered":"2024-04-25T14:02:14.620173+0000","last_clean":"2024-04-25T14:02:14.620173+0000","last_became_active":"2024-04-25T14:02:13.202452+0000","last_became_peered":"2024-04-25T14:02:13.202452+0000","last_unstale":"2024-04-25T14:02:14.620173+0000","last_undegraded":"2024-04-25T14:02:14.620173+0000","last_fullsized":"2024-04-25T14:02:14.620173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179693+0000","last_change":"2024-04-25T14:02:09.162133+0000","last_active":"2024-04-25T14:02:14.179693+0000","last_peered":"2024-04-25T14:02:14.179693+0000","last_clean":"2024-04-25T14:02:14.179693+0000","last_became_active":"2024-04-25T14:02:09.161795+0000","last_became_peered":"2024-04-25T14:02:09.161795+0000","last_unstale":"2024-04-25T14:02:14.179693+0000","last_undegraded":"2024-04-25T14:02:14.179693+0000","last_fullsized":"2024-04-25T14:02:14.179693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:49.968260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180083+0000","last_change":"2024-04-25T14:02:08.144670+0000","last_active":"2024-04-25T14:02:14.180083+0000","last_peered":"2024-04-25T14:02:14.180083+0000","last_clean":"2024-04-25T14:02:14.180083+0000","last_became_active":"2024-04-25T14:02:08.143303+0000","last_became_peered":"2024-04-25T14:02:08.143303+0000","last_unstale":"2024-04-25T14:02:14.180083+0000","last_undegraded":"2024-04-25T14:02:14.180083+0000","last_fullsized":"2024-04-25T14:02:14.180083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.199069+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.198521+0000","last_became_peered":"2024-04-25T14:02:13.198521+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.164561+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.164215+0000","last_became_peered":"2024-04-25T14:02:09.164215+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619618+0000","last_change":"2024-04-25T14:02:08.150157+0000","last_active":"2024-04-25T14:02:14.619618+0000","last_peered":"2024-04-25T14:02:14.619618+0000","last_clean":"2024-04-25T14:02:14.619618+0000","last_became_active":"2024-04-25T14:02:08.149812+0000","last_became_peered":"2024-04-25T14:02:08.149812+0000","last_unstale":"2024-04-25T14:02:14.619618+0000","last_undegraded":"2024-04-25T14:02:14.619618+0000","last_fullsized":"2024-04-25T14:02:14.619618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180256+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:02:14.180256+0000","last_peered":"2024-04-25T14:02:14.180256+0000","last_clean":"2024-04-25T14:02:14.180256+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:02:14.180256+0000","last_undegraded":"2024-04-25T14:02:14.180256+0000","last_fullsized":"2024-04-25T14:02:14.180256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.202985+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.202514+0000","last_became_peered":"2024-04-25T14:02:13.202514+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181284+0000","last_change":"2024-04-25T14:02:09.164697+0000","last_active":"2024-04-25T14:02:14.181284+0000","last_peered":"2024-04-25T14:02:14.181284+0000","last_clean":"2024-04-25T14:02:14.181284+0000","last_became_active":"2024-04-25T14:02:09.164352+0000","last_became_peered":"2024-04-25T14:02:09.164352+0000","last_unstale":"2024-04-25T14:02:14.181284+0000","last_undegraded":"2024-04-25T14:02:14.181284+0000","last_fullsized":"2024-04-25T14:02:14.181284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620427+0000","last_change":"2024-04-25T14:02:08.150437+0000","last_active":"2024-04-25T14:02:14.620427+0000","last_peered":"2024-04-25T14:02:14.620427+0000","last_clean":"2024-04-25T14:02:14.620427+0000","last_became_active":"2024-04-25T14:02:08.149954+0000","last_became_peered":"2024-04-25T14:02:08.149954+0000","last_unstale":"2024-04-25T14:02:14.620427+0000","last_undegraded":"2024-04-25T14:02:14.620427+0000","last_fullsized":"2024-04-25T14:02:14.620427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181863+0000","last_change":"2024-04-25T14:02:13.198918+0000","last_active":"2024-04-25T14:02:14.181863+0000","last_peered":"2024-04-25T14:02:14.181863+0000","last_clean":"2024-04-25T14:02:14.181863+0000","last_became_active":"2024-04-25T14:02:13.198244+0000","last_became_peered":"2024-04-25T14:02:13.198244+0000","last_unstale":"2024-04-25T14:02:14.181863+0000","last_undegraded":"2024-04-25T14:02:14.181863+0000","last_fullsized":"2024-04-25T14:02:14.181863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619706+0000","last_change":"2024-04-25T14:02:09.162899+0000","last_active":"2024-04-25T14:02:14.619706+0000","last_peered":"2024-04-25T14:02:14.619706+0000","last_clean":"2024-04-25T14:02:14.619706+0000","last_became_active":"2024-04-25T14:02:09.162573+0000","last_became_peered":"2024-04-25T14:02:09.162573+0000","last_unstale":"2024-04-25T14:02:14.619706+0000","last_undegraded":"2024-04-25T14:02:14.619706+0000","last_fullsized":"2024-04-25T14:02:14.619706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621027+0000","last_change":"2024-04-25T14:02:08.140157+0000","last_active":"2024-04-25T14:02:14.621027+0000","last_peered":"2024-04-25T14:02:14.621027+0000","last_clean":"2024-04-25T14:02:14.621027+0000","last_became_active":"2024-04-25T14:02:08.139477+0000","last_became_peered":"2024-04-25T14:02:08.139477+0000","last_unstale":"2024-04-25T14:02:14.621027+0000","last_undegraded":"2024-04-25T14:02:14.621027+0000","last_fullsized":"2024-04-25T14:02:14.621027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180180+0000","last_change":"2024-04-25T14:02:13.201803+0000","last_active":"2024-04-25T14:02:14.180180+0000","last_peered":"2024-04-25T14:02:14.180180+0000","last_clean":"2024-04-25T14:02:14.180180+0000","last_became_active":"2024-04-25T14:02:13.201388+0000","last_became_peered":"2024-04-25T14:02:13.201388+0000","last_unstale":"2024-04-25T14:02:14.180180+0000","last_undegraded":"2024-04-25T14:02:14.180180+0000","last_fullsized":"2024-04-25T14:02:14.180180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182344+0000","last_change":"2024-04-25T14:02:09.171757+0000","last_active":"2024-04-25T14:02:14.182344+0000","last_peered":"2024-04-25T14:02:14.182344+0000","last_clean":"2024-04-25T14:02:14.182344+0000","last_became_active":"2024-04-25T14:02:09.168908+0000","last_became_peered":"2024-04-25T14:02:09.168908+0000","last_unstale":"2024-04-25T14:02:14.182344+0000","last_undegraded":"2024-04-25T14:02:14.182344+0000","last_fullsized":"2024-04-25T14:02:14.182344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:21.339622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620710+0000","last_change":"2024-04-25T14:02:08.140886+0000","last_active":"2024-04-25T14:02:14.620710+0000","last_peered":"2024-04-25T14:02:14.620710+0000","last_clean":"2024-04-25T14:02:14.620710+0000","last_became_active":"2024-04-25T14:02:08.138585+0000","last_became_peered":"2024-04-25T14:02:08.138585+0000","last_unstale":"2024-04-25T14:02:14.620710+0000","last_undegraded":"2024-04-25T14:02:14.620710+0000","last_fullsized":"2024-04-25T14:02:14.620710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619005+0000","last_change":"2024-04-25T14:02:13.194033+0000","last_active":"2024-04-25T14:02:14.619005+0000","last_peered":"2024-04-25T14:02:14.619005+0000","last_clean":"2024-04-25T14:02:14.619005+0000","last_became_active":"2024-04-25T14:02:13.193486+0000","last_became_peered":"2024-04-25T14:02:13.193486+0000","last_unstale":"2024-04-25T14:02:14.619005+0000","last_undegraded":"2024-04-25T14:02:14.619005+0000","last_fullsized":"2024-04-25T14:02:14.619005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181242+0000","last_change":"2024-04-25T14:02:09.154176+0000","last_active":"2024-04-25T14:02:14.181242+0000","last_peered":"2024-04-25T14:02:14.181242+0000","last_clean":"2024-04-25T14:02:14.181242+0000","last_became_active":"2024-04-25T14:02:09.153867+0000","last_became_peered":"2024-04-25T14:02:09.153867+0000","last_unstale":"2024-04-25T14:02:14.181242+0000","last_undegraded":"2024-04-25T14:02:14.181242+0000","last_fullsized":"2024-04-25T14:02:14.181242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619639+0000","last_change":"2024-04-25T14:02:08.148162+0000","last_active":"2024-04-25T14:02:14.619639+0000","last_peered":"2024-04-25T14:02:14.619639+0000","last_clean":"2024-04-25T14:02:14.619639+0000","last_became_active":"2024-04-25T14:02:08.147822+0000","last_became_peered":"2024-04-25T14:02:08.147822+0000","last_unstale":"2024-04-25T14:02:14.619639+0000","last_undegraded":"2024-04-25T14:02:14.619639+0000","last_fullsized":"2024-04-25T14:02:14.619639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:47.001110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619644+0000","last_change":"2024-04-25T14:02:13.204464+0000","last_active":"2024-04-25T14:02:14.619644+0000","last_peered":"2024-04-25T14:02:14.619644+0000","last_clean":"2024-04-25T14:02:14.619644+0000","last_became_active":"2024-04-25T14:02:13.203501+0000","last_became_peered":"2024-04-25T14:02:13.203501+0000","last_unstale":"2024-04-25T14:02:14.619644+0000","last_undegraded":"2024-04-25T14:02:14.619644+0000","last_fullsized":"2024-04-25T14:02:14.619644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:36.450612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180471+0000","last_change":"2024-04-25T14:02:09.169427+0000","last_active":"2024-04-25T14:02:14.180471+0000","last_peered":"2024-04-25T14:02:14.180471+0000","last_clean":"2024-04-25T14:02:14.180471+0000","last_became_active":"2024-04-25T14:02:09.168920+0000","last_became_peered":"2024-04-25T14:02:09.168920+0000","last_unstale":"2024-04-25T14:02:14.180471+0000","last_undegraded":"2024-04-25T14:02:14.180471+0000","last_fullsized":"2024-04-25T14:02:14.180471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618690+0000","last_change":"2024-04-25T14:02:08.143121+0000","last_active":"2024-04-25T14:02:14.618690+0000","last_peered":"2024-04-25T14:02:14.618690+0000","last_clean":"2024-04-25T14:02:14.618690+0000","last_became_active":"2024-04-25T14:02:08.142765+0000","last_became_peered":"2024-04-25T14:02:08.142765+0000","last_unstale":"2024-04-25T14:02:14.618690+0000","last_undegraded":"2024-04-25T14:02:14.618690+0000","last_fullsized":"2024-04-25T14:02:14.618690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180517+0000","last_change":"2024-04-25T14:02:13.205101+0000","last_active":"2024-04-25T14:02:14.180517+0000","last_peered":"2024-04-25T14:02:14.180517+0000","last_clean":"2024-04-25T14:02:14.180517+0000","last_became_active":"2024-04-25T14:02:13.204696+0000","last_became_peered":"2024-04-25T14:02:13.204696+0000","last_unstale":"2024-04-25T14:02:14.180517+0000","last_undegraded":"2024-04-25T14:02:14.180517+0000","last_fullsized":"2024-04-25T14:02:14.180517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179895+0000","last_change":"2024-04-25T14:02:09.165439+0000","last_active":"2024-04-25T14:02:14.179895+0000","last_peered":"2024-04-25T14:02:14.179895+0000","last_clean":"2024-04-25T14:02:14.179895+0000","last_became_active":"2024-04-25T14:02:09.164934+0000","last_became_peered":"2024-04-25T14:02:09.164934+0000","last_unstale":"2024-04-25T14:02:14.179895+0000","last_undegraded":"2024-04-25T14:02:14.179895+0000","last_fullsized":"2024-04-25T14:02:14.179895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180810+0000","last_change":"2024-04-25T14:02:08.144013+0000","last_active":"2024-04-25T14:02:14.180810+0000","last_peered":"2024-04-25T14:02:14.180810+0000","last_clean":"2024-04-25T14:02:14.180810+0000","last_became_active":"2024-04-25T14:02:08.141723+0000","last_became_peered":"2024-04-25T14:02:08.141723+0000","last_unstale":"2024-04-25T14:02:14.180810+0000","last_undegraded":"2024-04-25T14:02:14.180810+0000","last_fullsized":"2024-04-25T14:02:14.180810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620974+0000","last_change":"2024-04-25T14:02:13.199511+0000","last_active":"2024-04-25T14:02:14.620974+0000","last_peered":"2024-04-25T14:02:14.620974+0000","last_clean":"2024-04-25T14:02:14.620974+0000","last_became_active":"2024-04-25T14:02:13.199074+0000","last_became_peered":"2024-04-25T14:02:13.199074+0000","last_unstale":"2024-04-25T14:02:14.620974+0000","last_undegraded":"2024-04-25T14:02:14.620974+0000","last_fullsized":"2024-04-25T14:02:14.620974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:47.660769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180811+0000","last_change":"2024-04-25T14:02:09.171097+0000","last_active":"2024-04-25T14:02:14.180811+0000","last_peered":"2024-04-25T14:02:14.180811+0000","last_clean":"2024-04-25T14:02:14.180811+0000","last_became_active":"2024-04-25T14:02:09.169656+0000","last_became_peered":"2024-04-25T14:02:09.169656+0000","last_unstale":"2024-04-25T14:02:14.180811+0000","last_undegraded":"2024-04-25T14:02:14.180811+0000","last_fullsized":"2024-04-25T14:02:14.180811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:34.914414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619583+0000","last_change":"2024-04-25T14:02:08.148427+0000","last_active":"2024-04-25T14:02:14.619583+0000","last_peered":"2024-04-25T14:02:14.619583+0000","last_clean":"2024-04-25T14:02:14.619583+0000","last_became_active":"2024-04-25T14:02:08.148019+0000","last_became_peered":"2024-04-25T14:02:08.148019+0000","last_unstale":"2024-04-25T14:02:14.619583+0000","last_undegraded":"2024-04-25T14:02:14.619583+0000","last_fullsized":"2024-04-25T14:02:14.619583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621238+0000","last_change":"2024-04-25T14:02:13.203865+0000","last_active":"2024-04-25T14:02:14.621238+0000","last_peered":"2024-04-25T14:02:14.621238+0000","last_clean":"2024-04-25T14:02:14.621238+0000","last_became_active":"2024-04-25T14:02:13.203056+0000","last_became_peered":"2024-04-25T14:02:13.203056+0000","last_unstale":"2024-04-25T14:02:14.621238+0000","last_undegraded":"2024-04-25T14:02:14.621238+0000","last_fullsized":"2024-04-25T14:02:14.621238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:58.555213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179953+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.179953+0000","last_peered":"2024-04-25T14:02:14.179953+0000","last_clean":"2024-04-25T14:02:14.179953+0000","last_became_active":"2024-04-25T14:02:09.171025+0000","last_became_peered":"2024-04-25T14:02:09.171025+0000","last_unstale":"2024-04-25T14:02:14.179953+0000","last_undegraded":"2024-04-25T14:02:14.179953+0000","last_fullsized":"2024-04-25T14:02:14.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180403+0000","last_change":"2024-04-25T14:02:08.141709+0000","last_active":"2024-04-25T14:02:14.180403+0000","last_peered":"2024-04-25T14:02:14.180403+0000","last_clean":"2024-04-25T14:02:14.180403+0000","last_became_active":"2024-04-25T14:02:08.141389+0000","last_became_peered":"2024-04-25T14:02:08.141389+0000","last_unstale":"2024-04-25T14:02:14.180403+0000","last_undegraded":"2024-04-25T14:02:14.180403+0000","last_fullsized":"2024-04-25T14:02:14.180403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620299+0000","last_change":"2024-04-25T14:02:13.205138+0000","last_active":"2024-04-25T14:02:14.620299+0000","last_peered":"2024-04-25T14:02:14.620299+0000","last_clean":"2024-04-25T14:02:14.620299+0000","last_became_active":"2024-04-25T14:02:13.204712+0000","last_became_peered":"2024-04-25T14:02:13.204712+0000","last_unstale":"2024-04-25T14:02:14.620299+0000","last_undegraded":"2024-04-25T14:02:14.620299+0000","last_fullsized":"2024-04-25T14:02:14.620299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181162+0000","last_change":"2024-04-25T14:02:09.159739+0000","last_active":"2024-04-25T14:02:14.181162+0000","last_peered":"2024-04-25T14:02:14.181162+0000","last_clean":"2024-04-25T14:02:14.181162+0000","last_became_active":"2024-04-25T14:02:09.158156+0000","last_became_peered":"2024-04-25T14:02:09.158156+0000","last_unstale":"2024-04-25T14:02:14.181162+0000","last_undegraded":"2024-04-25T14:02:14.181162+0000","last_fullsized":"2024-04-25T14:02:14.181162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:18.690850+0000","last_change":"2024-04-25T14:02:08.143908+0000","last_active":"2024-04-25T14:02:18.690850+0000","last_peered":"2024-04-25T14:02:18.690850+0000","last_clean":"2024-04-25T14:02:18.690850+0000","last_became_active":"2024-04-25T14:02:08.143478+0000","last_became_peered":"2024-04-25T14:02:08.143478+0000","last_unstale":"2024-04-25T14:02:18.690850+0000","last_undegraded":"2024-04-25T14:02:18.690850+0000","last_fullsized":"2024-04-25T14:02:18.690850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619926+0000","last_change":"2024-04-25T14:02:13.197367+0000","last_active":"2024-04-25T14:02:14.619926+0000","last_peered":"2024-04-25T14:02:14.619926+0000","last_clean":"2024-04-25T14:02:14.619926+0000","last_became_active":"2024-04-25T14:02:13.197029+0000","last_became_peered":"2024-04-25T14:02:13.197029+0000","last_unstale":"2024-04-25T14:02:14.619926+0000","last_undegraded":"2024-04-25T14:02:14.619926+0000","last_fullsized":"2024-04-25T14:02:14.619926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620116+0000","last_change":"2024-04-25T14:02:09.162706+0000","last_active":"2024-04-25T14:02:14.620116+0000","last_peered":"2024-04-25T14:02:14.620116+0000","last_clean":"2024-04-25T14:02:14.620116+0000","last_became_active":"2024-04-25T14:02:09.162316+0000","last_became_peered":"2024-04-25T14:02:09.162316+0000","last_unstale":"2024-04-25T14:02:14.620116+0000","last_undegraded":"2024-04-25T14:02:14.620116+0000","last_fullsized":"2024-04-25T14:02:14.620116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:23.993603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620975+0000","last_change":"2024-04-25T14:02:08.150406+0000","last_active":"2024-04-25T14:02:14.620975+0000","last_peered":"2024-04-25T14:02:14.620975+0000","last_clean":"2024-04-25T14:02:14.620975+0000","last_became_active":"2024-04-25T14:02:08.150057+0000","last_became_peered":"2024-04-25T14:02:08.150057+0000","last_unstale":"2024-04-25T14:02:14.620975+0000","last_undegraded":"2024-04-25T14:02:14.620975+0000","last_fullsized":"2024-04-25T14:02:14.620975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619602+0000","last_change":"2024-04-25T14:02:13.197851+0000","last_active":"2024-04-25T14:02:14.619602+0000","last_peered":"2024-04-25T14:02:14.619602+0000","last_clean":"2024-04-25T14:02:14.619602+0000","last_became_active":"2024-04-25T14:02:13.197109+0000","last_became_peered":"2024-04-25T14:02:13.197109+0000","last_unstale":"2024-04-25T14:02:14.619602+0000","last_undegraded":"2024-04-25T14:02:14.619602+0000","last_fullsized":"2024-04-25T14:02:14.619602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:20.798044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180425+0000","last_change":"2024-04-25T14:02:09.168810+0000","last_active":"2024-04-25T14:02:14.180425+0000","last_peered":"2024-04-25T14:02:14.180425+0000","last_clean":"2024-04-25T14:02:14.180425+0000","last_became_active":"2024-04-25T14:02:09.168323+0000","last_became_peered":"2024-04-25T14:02:09.168323+0000","last_unstale":"2024-04-25T14:02:14.180425+0000","last_undegraded":"2024-04-25T14:02:14.180425+0000","last_fullsized":"2024-04-25T14:02:14.180425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619958+0000","last_change":"2024-04-25T14:02:08.145115+0000","last_active":"2024-04-25T14:02:14.619958+0000","last_peered":"2024-04-25T14:02:14.619958+0000","last_clean":"2024-04-25T14:02:14.619958+0000","last_became_active":"2024-04-25T14:02:08.144853+0000","last_became_peered":"2024-04-25T14:02:08.144853+0000","last_unstale":"2024-04-25T14:02:14.619958+0000","last_undegraded":"2024-04-25T14:02:14.619958+0000","last_fullsized":"2024-04-25T14:02:14.619958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619831+0000","last_change":"2024-04-25T14:02:13.203422+0000","last_active":"2024-04-25T14:02:14.619831+0000","last_peered":"2024-04-25T14:02:14.619831+0000","last_clean":"2024-04-25T14:02:14.619831+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619831+0000","last_undegraded":"2024-04-25T14:02:14.619831+0000","last_fullsized":"2024-04-25T14:02:14.619831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619921+0000","last_change":"2024-04-25T14:02:09.171494+0000","last_active":"2024-04-25T14:02:14.619921+0000","last_peered":"2024-04-25T14:02:14.619921+0000","last_clean":"2024-04-25T14:02:14.619921+0000","last_became_active":"2024-04-25T14:02:09.171163+0000","last_became_peered":"2024-04-25T14:02:09.171163+0000","last_unstale":"2024-04-25T14:02:14.619921+0000","last_undegraded":"2024-04-25T14:02:14.619921+0000","last_fullsized":"2024-04-25T14:02:14.619921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181248+0000","last_change":"2024-04-25T14:02:08.150536+0000","last_active":"2024-04-25T14:02:14.181248+0000","last_peered":"2024-04-25T14:02:14.181248+0000","last_clean":"2024-04-25T14:02:14.181248+0000","last_became_active":"2024-04-25T14:02:08.150156+0000","last_became_peered":"2024-04-25T14:02:08.150156+0000","last_unstale":"2024-04-25T14:02:14.181248+0000","last_undegraded":"2024-04-25T14:02:14.181248+0000","last_fullsized":"2024-04-25T14:02:14.181248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619292+0000","last_change":"2024-04-25T14:02:13.201000+0000","last_active":"2024-04-25T14:02:14.619292+0000","last_peered":"2024-04-25T14:02:14.619292+0000","last_clean":"2024-04-25T14:02:14.619292+0000","last_became_active":"2024-04-25T14:02:13.200594+0000","last_became_peered":"2024-04-25T14:02:13.200594+0000","last_unstale":"2024-04-25T14:02:14.619292+0000","last_undegraded":"2024-04-25T14:02:14.619292+0000","last_fullsized":"2024-04-25T14:02:14.619292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.706373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180405+0000","last_change":"2024-04-25T14:02:13.204286+0000","last_active":"2024-04-25T14:02:14.180405+0000","last_peered":"2024-04-25T14:02:14.180405+0000","last_clean":"2024-04-25T14:02:14.180405+0000","last_became_active":"2024-04-25T14:02:13.203885+0000","last_became_peered":"2024-04-25T14:02:13.203885+0000","last_unstale":"2024-04-25T14:02:14.180405+0000","last_undegraded":"2024-04-25T14:02:14.180405+0000","last_fullsized":"2024-04-25T14:02:14.180405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619363+0000","last_change":"2024-04-25T14:02:09.158647+0000","last_active":"2024-04-25T14:02:14.619363+0000","last_peered":"2024-04-25T14:02:14.619363+0000","last_clean":"2024-04-25T14:02:14.619363+0000","last_became_active":"2024-04-25T14:02:09.158307+0000","last_became_peered":"2024-04-25T14:02:09.158307+0000","last_unstale":"2024-04-25T14:02:14.619363+0000","last_undegraded":"2024-04-25T14:02:14.619363+0000","last_fullsized":"2024-04-25T14:02:14.619363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619239+0000","last_change":"2024-04-25T14:02:08.149002+0000","last_active":"2024-04-25T14:02:14.619239+0000","last_peered":"2024-04-25T14:02:14.619239+0000","last_clean":"2024-04-25T14:02:14.619239+0000","last_became_active":"2024-04-25T14:02:08.148700+0000","last_became_peered":"2024-04-25T14:02:08.148700+0000","last_unstale":"2024-04-25T14:02:14.619239+0000","last_undegraded":"2024-04-25T14:02:14.619239+0000","last_fullsized":"2024-04-25T14:02:14.619239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181096+0000","last_change":"2024-04-25T14:02:13.197649+0000","last_active":"2024-04-25T14:02:14.181096+0000","last_peered":"2024-04-25T14:02:14.181096+0000","last_clean":"2024-04-25T14:02:14.181096+0000","last_became_active":"2024-04-25T14:02:13.197203+0000","last_became_peered":"2024-04-25T14:02:13.197203+0000","last_unstale":"2024-04-25T14:02:14.181096+0000","last_undegraded":"2024-04-25T14:02:14.181096+0000","last_fullsized":"2024-04-25T14:02:14.181096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619786+0000","last_change":"2024-04-25T14:02:09.164726+0000","last_active":"2024-04-25T14:02:14.619786+0000","last_peered":"2024-04-25T14:02:14.619786+0000","last_clean":"2024-04-25T14:02:14.619786+0000","last_became_active":"2024-04-25T14:02:09.164047+0000","last_became_peered":"2024-04-25T14:02:09.164047+0000","last_unstale":"2024-04-25T14:02:14.619786+0000","last_undegraded":"2024-04-25T14:02:14.619786+0000","last_fullsized":"2024-04-25T14:02:14.619786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619586+0000","last_change":"2024-04-25T14:02:08.138596+0000","last_active":"2024-04-25T14:02:14.619586+0000","last_peered":"2024-04-25T14:02:14.619586+0000","last_clean":"2024-04-25T14:02:14.619586+0000","last_became_active":"2024-04-25T14:02:08.138268+0000","last_became_peered":"2024-04-25T14:02:08.138268+0000","last_unstale":"2024-04-25T14:02:14.619586+0000","last_undegraded":"2024-04-25T14:02:14.619586+0000","last_fullsized":"2024-04-25T14:02:14.619586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182061+0000","last_change":"2024-04-25T14:02:08.144180+0000","last_active":"2024-04-25T14:02:14.182061+0000","last_peered":"2024-04-25T14:02:14.182061+0000","last_clean":"2024-04-25T14:02:14.182061+0000","last_became_active":"2024-04-25T14:02:08.142554+0000","last_became_peered":"2024-04-25T14:02:08.142554+0000","last_unstale":"2024-04-25T14:02:14.182061+0000","last_undegraded":"2024-04-25T14:02:14.182061+0000","last_fullsized":"2024-04-25T14:02:14.182061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.163851+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.163561+0000","last_became_peered":"2024-04-25T14:02:09.163561+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620647+0000","last_change":"2024-04-25T14:02:13.200670+0000","last_active":"2024-04-25T14:02:14.620647+0000","last_peered":"2024-04-25T14:02:14.620647+0000","last_clean":"2024-04-25T14:02:14.620647+0000","last_became_active":"2024-04-25T14:02:13.200255+0000","last_became_peered":"2024-04-25T14:02:13.200255+0000","last_unstale":"2024-04-25T14:02:14.620647+0000","last_undegraded":"2024-04-25T14:02:14.620647+0000","last_fullsized":"2024-04-25T14:02:14.620647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:49.451815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620669+0000","last_change":"2024-04-25T14:02:08.136214+0000","last_active":"2024-04-25T14:02:14.620669+0000","last_peered":"2024-04-25T14:02:14.620669+0000","last_clean":"2024-04-25T14:02:14.620669+0000","last_became_active":"2024-04-25T14:02:08.135892+0000","last_became_peered":"2024-04-25T14:02:08.135892+0000","last_unstale":"2024-04-25T14:02:14.620669+0000","last_undegraded":"2024-04-25T14:02:14.620669+0000","last_fullsized":"2024-04-25T14:02:14.620669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:00.331661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.160067+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.159705+0000","last_became_peered":"2024-04-25T14:02:09.159705+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:21.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621148+0000","last_change":"2024-04-25T14:02:13.203327+0000","last_active":"2024-04-25T14:02:14.621148+0000","last_peered":"2024-04-25T14:02:14.621148+0000","last_clean":"2024-04-25T14:02:14.621148+0000","last_became_active":"2024-04-25T14:02:13.202870+0000","last_became_peered":"2024-04-25T14:02:13.202870+0000","last_unstale":"2024-04-25T14:02:14.621148+0000","last_undegraded":"2024-04-25T14:02:14.621148+0000","last_fullsized":"2024-04-25T14:02:14.621148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:39.636544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179770+0000","last_change":"2024-04-25T14:02:09.165255+0000","last_active":"2024-04-25T14:02:14.179770+0000","last_peered":"2024-04-25T14:02:14.179770+0000","last_clean":"2024-04-25T14:02:14.179770+0000","last_became_active":"2024-04-25T14:02:09.164886+0000","last_became_peered":"2024-04-25T14:02:09.164886+0000","last_unstale":"2024-04-25T14:02:14.179770+0000","last_undegraded":"2024-04-25T14:02:14.179770+0000","last_fullsized":"2024-04-25T14:02:14.179770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619161+0000","last_change":"2024-04-25T14:02:09.161880+0000","last_active":"2024-04-25T14:02:14.619161+0000","last_peered":"2024-04-25T14:02:14.619161+0000","last_clean":"2024-04-25T14:02:14.619161+0000","last_became_active":"2024-04-25T14:02:09.161568+0000","last_became_peered":"2024-04-25T14:02:09.161568+0000","last_unstale":"2024-04-25T14:02:14.619161+0000","last_undegraded":"2024-04-25T14:02:14.619161+0000","last_fullsized":"2024-04-25T14:02:14.619161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619443+0000","last_change":"2024-04-25T14:02:09.163237+0000","last_active":"2024-04-25T14:02:14.619443+0000","last_peered":"2024-04-25T14:02:14.619443+0000","last_clean":"2024-04-25T14:02:14.619443+0000","last_became_active":"2024-04-25T14:02:09.162870+0000","last_became_peered":"2024-04-25T14:02:09.162870+0000","last_unstale":"2024-04-25T14:02:14.619443+0000","last_undegraded":"2024-04-25T14:02:14.619443+0000","last_fullsized":"2024-04-25T14:02:14.619443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180151+0000","last_change":"2024-04-25T14:02:09.167689+0000","last_active":"2024-04-25T14:02:14.180151+0000","last_peered":"2024-04-25T14:02:14.180151+0000","last_clean":"2024-04-25T14:02:14.180151+0000","last_became_active":"2024-04-25T14:02:09.167379+0000","last_became_peered":"2024-04-25T14:02:09.167379+0000","last_unstale":"2024-04-25T14:02:14.180151+0000","last_undegraded":"2024-04-25T14:02:14.180151+0000","last_fullsized":"2024-04-25T14:02:14.180151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621304+0000","last_change":"2024-04-25T14:02:09.155156+0000","last_active":"2024-04-25T14:02:14.621304+0000","last_peered":"2024-04-25T14:02:14.621304+0000","last_clean":"2024-04-25T14:02:14.621304+0000","last_became_active":"2024-04-25T14:02:09.154497+0000","last_became_peered":"2024-04-25T14:02:09.154497+0000","last_unstale":"2024-04-25T14:02:14.621304+0000","last_undegraded":"2024-04-25T14:02:14.621304+0000","last_fullsized":"2024-04-25T14:02:14.621304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620178+0000","last_change":"2024-04-25T14:02:09.161627+0000","last_active":"2024-04-25T14:02:14.620178+0000","last_peered":"2024-04-25T14:02:14.620178+0000","last_clean":"2024-04-25T14:02:14.620178+0000","last_became_active":"2024-04-25T14:02:09.161321+0000","last_became_peered":"2024-04-25T14:02:09.161321+0000","last_unstale":"2024-04-25T14:02:14.620178+0000","last_undegraded":"2024-04-25T14:02:14.620178+0000","last_fullsized":"2024-04-25T14:02:14.620178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181400+0000","last_change":"2024-04-25T14:02:09.177833+0000","last_active":"2024-04-25T14:02:14.181400+0000","last_peered":"2024-04-25T14:02:14.181400+0000","last_clean":"2024-04-25T14:02:14.181400+0000","last_became_active":"2024-04-25T14:02:09.177530+0000","last_became_peered":"2024-04-25T14:02:09.177530+0000","last_unstale":"2024-04-25T14:02:14.181400+0000","last_undegraded":"2024-04-25T14:02:14.181400+0000","last_fullsized":"2024-04-25T14:02:14.181400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621026+0000","last_change":"2024-04-25T14:02:09.160341+0000","last_active":"2024-04-25T14:02:14.621026+0000","last_peered":"2024-04-25T14:02:14.621026+0000","last_clean":"2024-04-25T14:02:14.621026+0000","last_became_active":"2024-04-25T14:02:09.160028+0000","last_became_peered":"2024-04-25T14:02:09.160028+0000","last_unstale":"2024-04-25T14:02:14.621026+0000","last_undegraded":"2024-04-25T14:02:14.621026+0000","last_fullsized":"2024-04-25T14:02:14.621026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619081+0000","last_change":"2024-04-25T14:02:09.163936+0000","last_active":"2024-04-25T14:02:14.619081+0000","last_peered":"2024-04-25T14:02:14.619081+0000","last_clean":"2024-04-25T14:02:14.619081+0000","last_became_active":"2024-04-25T14:02:09.163303+0000","last_became_peered":"2024-04-25T14:02:09.163303+0000","last_unstale":"2024-04-25T14:02:14.619081+0000","last_undegraded":"2024-04-25T14:02:14.619081+0000","last_fullsized":"2024-04-25T14:02:14.619081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619413+0000","last_change":"2024-04-25T14:02:09.154839+0000","last_active":"2024-04-25T14:02:14.619413+0000","last_peered":"2024-04-25T14:02:14.619413+0000","last_clean":"2024-04-25T14:02:14.619413+0000","last_became_active":"2024-04-25T14:02:09.154443+0000","last_became_peered":"2024-04-25T14:02:09.154443+0000","last_unstale":"2024-04-25T14:02:14.619413+0000","last_undegraded":"2024-04-25T14:02:14.619413+0000","last_fullsized":"2024-04-25T14:02:14.619413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619286+0000","last_change":"2024-04-25T14:02:09.171782+0000","last_active":"2024-04-25T14:02:14.619286+0000","last_peered":"2024-04-25T14:02:14.619286+0000","last_clean":"2024-04-25T14:02:14.619286+0000","last_became_active":"2024-04-25T14:02:09.171158+0000","last_became_peered":"2024-04-25T14:02:09.171158+0000","last_unstale":"2024-04-25T14:02:14.619286+0000","last_undegraded":"2024-04-25T14:02:14.619286+0000","last_fullsized":"2024-04-25T14:02:14.619286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179956+0000","last_change":"2024-04-25T14:02:09.177793+0000","last_active":"2024-04-25T14:02:14.179956+0000","last_peered":"2024-04-25T14:02:14.179956+0000","last_clean":"2024-04-25T14:02:14.179956+0000","last_became_active":"2024-04-25T14:02:09.177475+0000","last_became_peered":"2024-04-25T14:02:09.177475+0000","last_unstale":"2024-04-25T14:02:14.179956+0000","last_undegraded":"2024-04-25T14:02:14.179956+0000","last_fullsized":"2024-04-25T14:02:14.179956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181451+0000","last_change":"2024-04-25T14:02:09.166782+0000","last_active":"2024-04-25T14:02:14.181451+0000","last_peered":"2024-04-25T14:02:14.181451+0000","last_clean":"2024-04-25T14:02:14.181451+0000","last_became_active":"2024-04-25T14:02:09.166277+0000","last_became_peered":"2024-04-25T14:02:09.166277+0000","last_unstale":"2024-04-25T14:02:14.181451+0000","last_undegraded":"2024-04-25T14:02:14.181451+0000","last_fullsized":"2024-04-25T14:02:14.181451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181472+0000","last_change":"2024-04-25T14:02:09.167448+0000","last_active":"2024-04-25T14:02:14.181472+0000","last_peered":"2024-04-25T14:02:14.181472+0000","last_clean":"2024-04-25T14:02:14.181472+0000","last_became_active":"2024-04-25T14:02:09.167136+0000","last_became_peered":"2024-04-25T14:02:09.167136+0000","last_unstale":"2024-04-25T14:02:14.181472+0000","last_undegraded":"2024-04-25T14:02:14.181472+0000","last_fullsized":"2024-04-25T14:02:14.181472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620713+0000","last_change":"2024-04-25T14:02:09.167572+0000","last_active":"2024-04-25T14:02:14.620713+0000","last_peered":"2024-04-25T14:02:14.620713+0000","last_clean":"2024-04-25T14:02:14.620713+0000","last_became_active":"2024-04-25T14:02:09.167227+0000","last_became_peered":"2024-04-25T14:02:09.167227+0000","last_unstale":"2024-04-25T14:02:14.620713+0000","last_undegraded":"2024-04-25T14:02:14.620713+0000","last_fullsized":"2024-04-25T14:02:14.620713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620673+0000","last_change":"2024-04-25T14:02:09.174321+0000","last_active":"2024-04-25T14:02:14.620673+0000","last_peered":"2024-04-25T14:02:14.620673+0000","last_clean":"2024-04-25T14:02:14.620673+0000","last_became_active":"2024-04-25T14:02:09.174015+0000","last_became_peered":"2024-04-25T14:02:09.174015+0000","last_unstale":"2024-04-25T14:02:14.620673+0000","last_undegraded":"2024-04-25T14:02:14.620673+0000","last_fullsized":"2024-04-25T14:02:14.620673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618957+0000","last_change":"2024-04-25T14:02:09.169554+0000","last_active":"2024-04-25T14:02:14.618957+0000","last_peered":"2024-04-25T14:02:14.618957+0000","last_clean":"2024-04-25T14:02:14.618957+0000","last_became_active":"2024-04-25T14:02:09.168555+0000","last_became_peered":"2024-04-25T14:02:09.168555+0000","last_unstale":"2024-04-25T14:02:14.618957+0000","last_undegraded":"2024-04-25T14:02:14.618957+0000","last_fullsized":"2024-04-25T14:02:14.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619417+0000","last_change":"2024-04-25T14:02:09.160170+0000","last_active":"2024-04-25T14:02:14.619417+0000","last_peered":"2024-04-25T14:02:14.619417+0000","last_clean":"2024-04-25T14:02:14.619417+0000","last_became_active":"2024-04-25T14:02:09.159871+0000","last_became_peered":"2024-04-25T14:02:09.159871+0000","last_unstale":"2024-04-25T14:02:14.619417+0000","last_undegraded":"2024-04-25T14:02:14.619417+0000","last_fullsized":"2024-04-25T14:02:14.619417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179980+0000","last_change":"2024-04-25T14:02:09.159887+0000","last_active":"2024-04-25T14:02:14.179980+0000","last_peered":"2024-04-25T14:02:14.179980+0000","last_clean":"2024-04-25T14:02:14.179980+0000","last_became_active":"2024-04-25T14:02:09.159431+0000","last_became_peered":"2024-04-25T14:02:09.159431+0000","last_unstale":"2024-04-25T14:02:14.179980+0000","last_undegraded":"2024-04-25T14:02:14.179980+0000","last_fullsized":"2024-04-25T14:02:14.179980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620557+0000","last_change":"2024-04-25T14:02:09.167290+0000","last_active":"2024-04-25T14:02:14.620557+0000","last_peered":"2024-04-25T14:02:14.620557+0000","last_clean":"2024-04-25T14:02:14.620557+0000","last_became_active":"2024-04-25T14:02:09.166974+0000","last_became_peered":"2024-04-25T14:02:09.166974+0000","last_unstale":"2024-04-25T14:02:14.620557+0000","last_undegraded":"2024-04-25T14:02:14.620557+0000","last_fullsized":"2024-04-25T14:02:14.620557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620085+0000","last_change":"2024-04-25T14:02:09.169185+0000","last_active":"2024-04-25T14:02:14.620085+0000","last_peered":"2024-04-25T14:02:14.620085+0000","last_clean":"2024-04-25T14:02:14.620085+0000","last_became_active":"2024-04-25T14:02:09.168862+0000","last_became_peered":"2024-04-25T14:02:09.168862+0000","last_unstale":"2024-04-25T14:02:14.620085+0000","last_undegraded":"2024-04-25T14:02:14.620085+0000","last_fullsized":"2024-04-25T14:02:14.620085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180614+0000","last_change":"2024-04-25T14:02:09.174006+0000","last_active":"2024-04-25T14:02:14.180614+0000","last_peered":"2024-04-25T14:02:14.180614+0000","last_clean":"2024-04-25T14:02:14.180614+0000","last_became_active":"2024-04-25T14:02:09.173662+0000","last_became_peered":"2024-04-25T14:02:09.173662+0000","last_unstale":"2024-04-25T14:02:14.180614+0000","last_undegraded":"2024-04-25T14:02:14.180614+0000","last_fullsized":"2024-04-25T14:02:14.180614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181263+0000","last_change":"2024-04-25T14:02:09.174331+0000","last_active":"2024-04-25T14:02:14.181263+0000","last_peered":"2024-04-25T14:02:14.181263+0000","last_clean":"2024-04-25T14:02:14.181263+0000","last_became_active":"2024-04-25T14:02:09.173998+0000","last_became_peered":"2024-04-25T14:02:09.173998+0000","last_unstale":"2024-04-25T14:02:14.181263+0000","last_undegraded":"2024-04-25T14:02:14.181263+0000","last_fullsized":"2024-04-25T14:02:14.181263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182263+0000","last_change":"2024-04-25T14:02:09.166767+0000","last_active":"2024-04-25T14:02:14.182263+0000","last_peered":"2024-04-25T14:02:14.182263+0000","last_clean":"2024-04-25T14:02:14.182263+0000","last_became_active":"2024-04-25T14:02:09.165131+0000","last_became_peered":"2024-04-25T14:02:09.165131+0000","last_unstale":"2024-04-25T14:02:14.182263+0000","last_undegraded":"2024-04-25T14:02:14.182263+0000","last_fullsized":"2024-04-25T14:02:14.182263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:07.386855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619681+0000","last_change":"2024-04-25T14:02:09.164960+0000","last_active":"2024-04-25T14:02:14.619681+0000","last_peered":"2024-04-25T14:02:14.619681+0000","last_clean":"2024-04-25T14:02:14.619681+0000","last_became_active":"2024-04-25T14:02:09.164417+0000","last_became_peered":"2024-04-25T14:02:09.164417+0000","last_unstale":"2024-04-25T14:02:14.619681+0000","last_undegraded":"2024-04-25T14:02:14.619681+0000","last_fullsized":"2024-04-25T14:02:14.619681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180423+0000","last_change":"2024-04-25T14:02:09.161177+0000","last_active":"2024-04-25T14:02:14.180423+0000","last_peered":"2024-04-25T14:02:14.180423+0000","last_clean":"2024-04-25T14:02:14.180423+0000","last_became_active":"2024-04-25T14:02:09.160404+0000","last_became_peered":"2024-04-25T14:02:09.160404+0000","last_unstale":"2024-04-25T14:02:14.180423+0000","last_undegraded":"2024-04-25T14:02:14.180423+0000","last_fullsized":"2024-04-25T14:02:14.180423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619241+0000","last_change":"2024-04-25T14:02:09.170498+0000","last_active":"2024-04-25T14:02:14.619241+0000","last_peered":"2024-04-25T14:02:14.619241+0000","last_clean":"2024-04-25T14:02:14.619241+0000","last_became_active":"2024-04-25T14:02:09.167994+0000","last_became_peered":"2024-04-25T14:02:09.167994+0000","last_unstale":"2024-04-25T14:02:14.619241+0000","last_undegraded":"2024-04-25T14:02:14.619241+0000","last_fullsized":"2024-04-25T14:02:14.619241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180565+0000","last_change":"2024-04-25T14:02:09.155470+0000","last_active":"2024-04-25T14:02:14.180565+0000","last_peered":"2024-04-25T14:02:14.180565+0000","last_clean":"2024-04-25T14:02:14.180565+0000","last_became_active":"2024-04-25T14:02:09.155123+0000","last_became_peered":"2024-04-25T14:02:09.155123+0000","last_unstale":"2024-04-25T14:02:14.180565+0000","last_undegraded":"2024-04-25T14:02:14.180565+0000","last_fullsized":"2024-04-25T14:02:14.180565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619577+0000","last_change":"2024-04-25T14:02:09.163082+0000","last_active":"2024-04-25T14:02:14.619577+0000","last_peered":"2024-04-25T14:02:14.619577+0000","last_clean":"2024-04-25T14:02:14.619577+0000","last_became_active":"2024-04-25T14:02:09.162148+0000","last_became_peered":"2024-04-25T14:02:09.162148+0000","last_unstale":"2024-04-25T14:02:14.619577+0000","last_undegraded":"2024-04-25T14:02:14.619577+0000","last_fullsized":"2024-04-25T14:02:14.619577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.167223+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.166924+0000","last_became_peered":"2024-04-25T14:02:09.166924+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619169+0000","last_change":"2024-04-25T14:02:09.159838+0000","last_active":"2024-04-25T14:02:14.619169+0000","last_peered":"2024-04-25T14:02:14.619169+0000","last_clean":"2024-04-25T14:02:14.619169+0000","last_became_active":"2024-04-25T14:02:09.159551+0000","last_became_peered":"2024-04-25T14:02:09.159551+0000","last_unstale":"2024-04-25T14:02:14.619169+0000","last_undegraded":"2024-04-25T14:02:14.619169+0000","last_fullsized":"2024-04-25T14:02:14.619169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182085+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.182085+0000","last_peered":"2024-04-25T14:02:14.182085+0000","last_clean":"2024-04-25T14:02:14.182085+0000","last_became_active":"2024-04-25T14:02:09.168711+0000","last_became_peered":"2024-04-25T14:02:09.168711+0000","last_unstale":"2024-04-25T14:02:14.182085+0000","last_undegraded":"2024-04-25T14:02:14.182085+0000","last_fullsized":"2024-04-25T14:02:14.182085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:02:32.244 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:02:32.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.386+0000 7ff2d8114640 1 Processor -- start 2024-04-25T14:02:32.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.386+0000 7ff2d8114640 1 -- start start 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.387+0000 7ff2d8114640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 0x7ff2d006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d8114640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 0x7ff2d00ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d8114640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2d00b4750 con 0x7ff2d006f780 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d8114640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff2d015e310 con 0x7ff2d00b5f00 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d8114640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d00b5360 con 0x7ff2d015df00 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d66b1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 0x7ff2d00ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d5eb0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 0x7ff2d006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d56af640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7ff2d006f780 0x7ff2d00b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:33110/0 (socket says 172.21.15.150:33110) 2024-04-25T14:02:32.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d56af640 1 -- 172.21.15.150:0/1580295768 learned_addr learned my addr 172.21.15.150:0/1580295768 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:32.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d4eae640 1 -- 172.21.15.150:0/1580295768 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4014471294 0 0) 0x7ff2d00b4750 con 0x7ff2d006f780 2024-04-25T14:02:32.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d4eae640 1 -- 172.21.15.150:0/1580295768 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2bc003520 con 0x7ff2d006f780 2024-04-25T14:02:32.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d5eb0640 1 -- 172.21.15.150:0/1580295768 >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 msgr2=0x7ff2d00ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:32.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d5eb0640 1 --2- 172.21.15.150:0/1580295768 >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 0x7ff2d00ae7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d5eb0640 1 -- 172.21.15.150:0/1580295768 >> v1:172.21.15.187:6789/0 conn(0x7ff2d006f780 legacy=0x7ff2d00b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d5eb0640 1 -- 172.21.15.150:0/1580295768 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d00b4a30 con 0x7ff2d015df00 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.388+0000 7ff2d66b1640 1 --2- 172.21.15.150:0/1580295768 >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 0x7ff2d00ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.389+0000 7ff2d5eb0640 1 --2- 172.21.15.150:0/1580295768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 0x7ff2d006f240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0097a0 tx=0x7ff2cc02a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d82a83893d9cd078 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.389+0000 7ff2d4eae640 1 -- 172.21.15.150:0/1580295768 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc037070 con 0x7ff2d015df00 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.389+0000 7ff2d4eae640 1 -- 172.21.15.150:0/1580295768 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2cc02ae50 con 0x7ff2d015df00 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.389+0000 7ff2d4eae640 1 -- 172.21.15.150:0/1580295768 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc032640 con 0x7ff2d015df00 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 -- 172.21.15.150:0/1580295768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 msgr2=0x7ff2d006f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 --2- 172.21.15.150:0/1580295768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 0x7ff2d006f240 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0097a0 tx=0x7ff2cc02a100 comp rx=0 tx=0).stop 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 -- 172.21.15.150:0/1580295768 shutdown_connections 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 --2- 172.21.15.150:0/1580295768 >> v2:172.21.15.187:3300/0 conn(0x7ff2d00b5f00 0x7ff2d00ae7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 --2- 172.21.15.150:0/1580295768 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d015df00 0x7ff2d006f240 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 -- 172.21.15.150:0/1580295768 >> 172.21.15.150:0/1580295768 conn(0x7ff2d00d1ae0 msgr2=0x7ff2d00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 -- 172.21.15.150:0/1580295768 shutdown_connections 2024-04-25T14:02:32.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.390+0000 7ff2d8114640 1 -- 172.21.15.150:0/1580295768 wait complete. 2024-04-25T14:02:32.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.392+0000 7ff2d8114640 1 Processor -- start 2024-04-25T14:02:32.392 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.392+0000 7ff2d8114640 1 -- start start 2024-04-25T14:02:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d006f780 0x7ff2d009bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2d015df00 0x7ff2d00cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d00aed20 con 0x7ff2d00b5f00 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d00b4ff0 con 0x7ff2d015df00 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d8114640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff2d00aeea0 con 0x7ff2d006f780 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d56af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d56af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38238/0 (socket says 172.21.15.150:38238) 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d56af640 1 -- 172.21.15.150:0/3074932094 learned_addr learned my addr 172.21.15.150:0/3074932094 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d66b1640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2d015df00 0x7ff2d00cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.393+0000 7ff2d5eb0640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d006f780 0x7ff2d009bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 -- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d006f780 msgr2=0x7ff2d009bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d006f780 0x7ff2d009bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 -- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2d015df00 msgr2=0x7ff2d00cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2d015df00 0x7ff2d00cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 -- 172.21.15.150:0/3074932094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d009e3f0 con 0x7ff2d00b5f00 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d56af640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000c3c0 tx=0x7ff2c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c0017070 con 0x7ff2d00b5f00 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c0005ce0 con 0x7ff2d00b5f00 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d009e6d0 con 0x7ff2d00b5f00 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c00072f0 con 0x7ff2d00b5f00 2024-04-25T14:02:32.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.394+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d009ec90 con 0x7ff2d00b5f00 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.395+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d00b6480 con 0x7ff2d00b5f00 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.396+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff2c001a070 con 0x7ff2d00b5f00 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.396+0000 7ff2c67fc640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 0x7ff2b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.396+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff2c00948b0 con 0x7ff2d00b5f00 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.396+0000 7ff2d5eb0640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 0x7ff2b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.397+0000 7ff2d5eb0640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 0x7ff2b40769c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc034000 tx=0x7ff2cc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.402+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff2d00b6480 con 0x7ff2d00b5f00 2024-04-25T14:02:32.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.572+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff2d009e170 con 0x7ff2b40744b0 2024-04-25T14:02:32.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.580+0000 7ff2c67fc640 1 -- 172.21.15.150:0/3074932094 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332523 (secure 0 0 0) 0x7ff2cc093300 con 0x7ff2b40744b0 2024-04-25T14:02:32.583 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:32.585 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 msgr2=0x7ff2b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 0x7ff2b40769c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc034000 tx=0x7ff2cc02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 msgr2=0x7ff2d0159180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff2c000c3c0 tx=0x7ff2c000c880 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 shutdown_connections 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff2b40744b0 0x7ff2b40769c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff2d015df00 0x7ff2d00cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff2d00b5f00 0x7ff2d0159180 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 --2- 172.21.15.150:0/3074932094 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff2d006f780 0x7ff2d009bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 >> 172.21.15.150:0/3074932094 conn(0x7ff2d00d1ae0 msgr2=0x7ff2d015c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 shutdown_connections 2024-04-25T14:02:32.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.583+0000 7ff2d8114640 1 -- 172.21.15.150:0/3074932094 wait complete. 2024-04-25T14:02:32.594 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-04-25T14:02:30.591173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003104"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180578+0000","last_change":"2024-04-25T14:02:08.147737+0000","last_active":"2024-04-25T14:02:14.180578+0000","last_peered":"2024-04-25T14:02:14.180578+0000","last_clean":"2024-04-25T14:02:14.180578+0000","last_became_active":"2024-04-25T14:02:08.147072+0000","last_became_peered":"2024-04-25T14:02:08.147072+0000","last_unstale":"2024-04-25T14:02:14.180578+0000","last_undegraded":"2024-04-25T14:02:14.180578+0000","last_fullsized":"2024-04-25T14:02:14.180578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621342+0000","last_change":"2024-04-25T14:02:09.168014+0000","last_active":"2024-04-25T14:02:14.621342+0000","last_peered":"2024-04-25T14:02:14.621342+0000","last_clean":"2024-04-25T14:02:14.621342+0000","last_became_active":"2024-04-25T14:02:09.166846+0000","last_became_peered":"2024-04-25T14:02:09.166846+0000","last_unstale":"2024-04-25T14:02:14.621342+0000","last_undegraded":"2024-04-25T14:02:14.621342+0000","last_fullsized":"2024-04-25T14:02:14.621342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619101+0000","last_change":"2024-04-25T14:02:13.203189+0000","last_active":"2024-04-25T14:02:14.619101+0000","last_peered":"2024-04-25T14:02:14.619101+0000","last_clean":"2024-04-25T14:02:14.619101+0000","last_became_active":"2024-04-25T14:02:13.201743+0000","last_became_peered":"2024-04-25T14:02:13.201743+0000","last_unstale":"2024-04-25T14:02:14.619101+0000","last_undegraded":"2024-04-25T14:02:14.619101+0000","last_fullsized":"2024-04-25T14:02:14.619101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621459+0000","last_change":"2024-04-25T14:02:08.147961+0000","last_active":"2024-04-25T14:02:14.621459+0000","last_peered":"2024-04-25T14:02:14.621459+0000","last_clean":"2024-04-25T14:02:14.621459+0000","last_became_active":"2024-04-25T14:02:08.147652+0000","last_became_peered":"2024-04-25T14:02:08.147652+0000","last_unstale":"2024-04-25T14:02:14.621459+0000","last_undegraded":"2024-04-25T14:02:14.621459+0000","last_fullsized":"2024-04-25T14:02:14.621459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180307+0000","last_change":"2024-04-25T14:02:09.173887+0000","last_active":"2024-04-25T14:02:14.180307+0000","last_peered":"2024-04-25T14:02:14.180307+0000","last_clean":"2024-04-25T14:02:14.180307+0000","last_became_active":"2024-04-25T14:02:09.173583+0000","last_became_peered":"2024-04-25T14:02:09.173583+0000","last_unstale":"2024-04-25T14:02:14.180307+0000","last_undegraded":"2024-04-25T14:02:14.180307+0000","last_fullsized":"2024-04-25T14:02:14.180307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619293+0000","last_change":"2024-04-25T14:02:13.202095+0000","last_active":"2024-04-25T14:02:14.619293+0000","last_peered":"2024-04-25T14:02:14.619293+0000","last_clean":"2024-04-25T14:02:14.619293+0000","last_became_active":"2024-04-25T14:02:13.201620+0000","last_became_peered":"2024-04-25T14:02:13.201620+0000","last_unstale":"2024-04-25T14:02:14.619293+0000","last_undegraded":"2024-04-25T14:02:14.619293+0000","last_fullsized":"2024-04-25T14:02:14.619293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:25.777824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181439+0000","last_change":"2024-04-25T14:02:08.139279+0000","last_active":"2024-04-25T14:02:14.181439+0000","last_peered":"2024-04-25T14:02:14.181439+0000","last_clean":"2024-04-25T14:02:14.181439+0000","last_became_active":"2024-04-25T14:02:08.138923+0000","last_became_peered":"2024-04-25T14:02:08.138923+0000","last_unstale":"2024-04-25T14:02:14.181439+0000","last_undegraded":"2024-04-25T14:02:14.181439+0000","last_fullsized":"2024-04-25T14:02:14.181439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:22.987870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619690+0000","last_change":"2024-04-25T14:02:09.162551+0000","last_active":"2024-04-25T14:02:14.619690+0000","last_peered":"2024-04-25T14:02:14.619690+0000","last_clean":"2024-04-25T14:02:14.619690+0000","last_became_active":"2024-04-25T14:02:09.161794+0000","last_became_peered":"2024-04-25T14:02:09.161794+0000","last_unstale":"2024-04-25T14:02:14.619690+0000","last_undegraded":"2024-04-25T14:02:14.619690+0000","last_fullsized":"2024-04-25T14:02:14.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619746+0000","last_change":"2024-04-25T14:02:13.203879+0000","last_active":"2024-04-25T14:02:14.619746+0000","last_peered":"2024-04-25T14:02:14.619746+0000","last_clean":"2024-04-25T14:02:14.619746+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619746+0000","last_undegraded":"2024-04-25T14:02:14.619746+0000","last_fullsized":"2024-04-25T14:02:14.619746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.794247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180468+0000","last_change":"2024-04-25T14:02:08.148407+0000","last_active":"2024-04-25T14:02:14.180468+0000","last_peered":"2024-04-25T14:02:14.180468+0000","last_clean":"2024-04-25T14:02:14.180468+0000","last_became_active":"2024-04-25T14:02:08.148049+0000","last_became_peered":"2024-04-25T14:02:08.148049+0000","last_unstale":"2024-04-25T14:02:14.180468+0000","last_undegraded":"2024-04-25T14:02:14.180468+0000","last_fullsized":"2024-04-25T14:02:14.180468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620378+0000","last_change":"2024-04-25T14:02:09.162054+0000","last_active":"2024-04-25T14:02:14.620378+0000","last_peered":"2024-04-25T14:02:14.620378+0000","last_clean":"2024-04-25T14:02:14.620378+0000","last_became_active":"2024-04-25T14:02:09.161561+0000","last_became_peered":"2024-04-25T14:02:09.161561+0000","last_unstale":"2024-04-25T14:02:14.620378+0000","last_undegraded":"2024-04-25T14:02:14.620378+0000","last_fullsized":"2024-04-25T14:02:14.620378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620224+0000","last_change":"2024-04-25T14:02:13.202231+0000","last_active":"2024-04-25T14:02:14.620224+0000","last_peered":"2024-04-25T14:02:14.620224+0000","last_clean":"2024-04-25T14:02:14.620224+0000","last_became_active":"2024-04-25T14:02:13.201805+0000","last_became_peered":"2024-04-25T14:02:13.201805+0000","last_unstale":"2024-04-25T14:02:14.620224+0000","last_undegraded":"2024-04-25T14:02:14.620224+0000","last_fullsized":"2024-04-25T14:02:14.620224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:12.379500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181919+0000","last_change":"2024-04-25T14:02:09.164630+0000","last_active":"2024-04-25T14:02:14.181919+0000","last_peered":"2024-04-25T14:02:14.181919+0000","last_clean":"2024-04-25T14:02:14.181919+0000","last_became_active":"2024-04-25T14:02:09.164292+0000","last_became_peered":"2024-04-25T14:02:09.164292+0000","last_unstale":"2024-04-25T14:02:14.181919+0000","last_undegraded":"2024-04-25T14:02:14.181919+0000","last_fullsized":"2024-04-25T14:02:14.181919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180969+0000","last_change":"2024-04-25T14:02:08.136088+0000","last_active":"2024-04-25T14:02:14.180969+0000","last_peered":"2024-04-25T14:02:14.180969+0000","last_clean":"2024-04-25T14:02:14.180969+0000","last_became_active":"2024-04-25T14:02:08.135716+0000","last_became_peered":"2024-04-25T14:02:08.135716+0000","last_unstale":"2024-04-25T14:02:14.180969+0000","last_undegraded":"2024-04-25T14:02:14.180969+0000","last_fullsized":"2024-04-25T14:02:14.180969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:27.040903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620573+0000","last_change":"2024-04-25T14:02:13.196592+0000","last_active":"2024-04-25T14:02:14.620573+0000","last_peered":"2024-04-25T14:02:14.620573+0000","last_clean":"2024-04-25T14:02:14.620573+0000","last_became_active":"2024-04-25T14:02:13.196120+0000","last_became_peered":"2024-04-25T14:02:13.196120+0000","last_unstale":"2024-04-25T14:02:14.620573+0000","last_undegraded":"2024-04-25T14:02:14.620573+0000","last_fullsized":"2024-04-25T14:02:14.620573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620568+0000","last_change":"2024-04-25T14:02:09.163625+0000","last_active":"2024-04-25T14:02:14.620568+0000","last_peered":"2024-04-25T14:02:14.620568+0000","last_clean":"2024-04-25T14:02:14.620568+0000","last_became_active":"2024-04-25T14:02:09.163320+0000","last_became_peered":"2024-04-25T14:02:09.163320+0000","last_unstale":"2024-04-25T14:02:14.620568+0000","last_undegraded":"2024-04-25T14:02:14.620568+0000","last_fullsized":"2024-04-25T14:02:14.620568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619947+0000","last_change":"2024-04-25T14:02:08.140552+0000","last_active":"2024-04-25T14:02:14.619947+0000","last_peered":"2024-04-25T14:02:14.619947+0000","last_clean":"2024-04-25T14:02:14.619947+0000","last_became_active":"2024-04-25T14:02:08.139997+0000","last_became_peered":"2024-04-25T14:02:08.139997+0000","last_unstale":"2024-04-25T14:02:14.619947+0000","last_undegraded":"2024-04-25T14:02:14.619947+0000","last_fullsized":"2024-04-25T14:02:14.619947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620055+0000","last_change":"2024-04-25T14:02:13.204879+0000","last_active":"2024-04-25T14:02:14.620055+0000","last_peered":"2024-04-25T14:02:14.620055+0000","last_clean":"2024-04-25T14:02:14.620055+0000","last_became_active":"2024-04-25T14:02:13.203934+0000","last_became_peered":"2024-04-25T14:02:13.203934+0000","last_unstale":"2024-04-25T14:02:14.620055+0000","last_undegraded":"2024-04-25T14:02:14.620055+0000","last_fullsized":"2024-04-25T14:02:14.620055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619890+0000","last_change":"2024-04-25T14:02:09.157409+0000","last_active":"2024-04-25T14:02:14.619890+0000","last_peered":"2024-04-25T14:02:14.619890+0000","last_clean":"2024-04-25T14:02:14.619890+0000","last_became_active":"2024-04-25T14:02:09.157102+0000","last_became_peered":"2024-04-25T14:02:09.157102+0000","last_unstale":"2024-04-25T14:02:14.619890+0000","last_undegraded":"2024-04-25T14:02:14.619890+0000","last_fullsized":"2024-04-25T14:02:14.619890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619463+0000","last_change":"2024-04-25T14:02:08.138257+0000","last_active":"2024-04-25T14:02:14.619463+0000","last_peered":"2024-04-25T14:02:14.619463+0000","last_clean":"2024-04-25T14:02:14.619463+0000","last_became_active":"2024-04-25T14:02:08.137925+0000","last_became_peered":"2024-04-25T14:02:08.137925+0000","last_unstale":"2024-04-25T14:02:14.619463+0000","last_undegraded":"2024-04-25T14:02:14.619463+0000","last_fullsized":"2024-04-25T14:02:14.619463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619999+0000","last_change":"2024-04-25T14:02:13.204203+0000","last_active":"2024-04-25T14:02:14.619999+0000","last_peered":"2024-04-25T14:02:14.619999+0000","last_clean":"2024-04-25T14:02:14.619999+0000","last_became_active":"2024-04-25T14:02:13.203578+0000","last_became_peered":"2024-04-25T14:02:13.203578+0000","last_unstale":"2024-04-25T14:02:14.619999+0000","last_undegraded":"2024-04-25T14:02:14.619999+0000","last_fullsized":"2024-04-25T14:02:14.619999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620454+0000","last_change":"2024-04-25T14:02:09.153035+0000","last_active":"2024-04-25T14:02:14.620454+0000","last_peered":"2024-04-25T14:02:14.620454+0000","last_clean":"2024-04-25T14:02:14.620454+0000","last_became_active":"2024-04-25T14:02:09.152071+0000","last_became_peered":"2024-04-25T14:02:09.152071+0000","last_unstale":"2024-04-25T14:02:14.620454+0000","last_undegraded":"2024-04-25T14:02:14.620454+0000","last_fullsized":"2024-04-25T14:02:14.620454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:11.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180075+0000","last_change":"2024-04-25T14:02:08.141740+0000","last_active":"2024-04-25T14:02:14.180075+0000","last_peered":"2024-04-25T14:02:14.180075+0000","last_clean":"2024-04-25T14:02:14.180075+0000","last_became_active":"2024-04-25T14:02:08.141400+0000","last_became_peered":"2024-04-25T14:02:08.141400+0000","last_unstale":"2024-04-25T14:02:14.180075+0000","last_undegraded":"2024-04-25T14:02:14.180075+0000","last_fullsized":"2024-04-25T14:02:14.180075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619288+0000","last_change":"2024-04-25T14:02:13.202697+0000","last_active":"2024-04-25T14:02:14.619288+0000","last_peered":"2024-04-25T14:02:14.619288+0000","last_clean":"2024-04-25T14:02:14.619288+0000","last_became_active":"2024-04-25T14:02:13.201555+0000","last_became_peered":"2024-04-25T14:02:13.201555+0000","last_unstale":"2024-04-25T14:02:14.619288+0000","last_undegraded":"2024-04-25T14:02:14.619288+0000","last_fullsized":"2024-04-25T14:02:14.619288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:43.769087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180683+0000","last_change":"2024-04-25T14:02:09.166496+0000","last_active":"2024-04-25T14:02:14.180683+0000","last_peered":"2024-04-25T14:02:14.180683+0000","last_clean":"2024-04-25T14:02:14.180683+0000","last_became_active":"2024-04-25T14:02:09.166180+0000","last_became_peered":"2024-04-25T14:02:09.166180+0000","last_unstale":"2024-04-25T14:02:14.180683+0000","last_undegraded":"2024-04-25T14:02:14.180683+0000","last_fullsized":"2024-04-25T14:02:14.180683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181386+0000","last_change":"2024-04-25T14:02:08.143988+0000","last_active":"2024-04-25T14:02:14.181386+0000","last_peered":"2024-04-25T14:02:14.181386+0000","last_clean":"2024-04-25T14:02:14.181386+0000","last_became_active":"2024-04-25T14:02:08.143639+0000","last_became_peered":"2024-04-25T14:02:08.143639+0000","last_unstale":"2024-04-25T14:02:14.181386+0000","last_undegraded":"2024-04-25T14:02:14.181386+0000","last_fullsized":"2024-04-25T14:02:14.181386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620239+0000","last_change":"2024-04-25T14:02:13.193130+0000","last_active":"2024-04-25T14:02:14.620239+0000","last_peered":"2024-04-25T14:02:14.620239+0000","last_clean":"2024-04-25T14:02:14.620239+0000","last_became_active":"2024-04-25T14:02:13.192664+0000","last_became_peered":"2024-04-25T14:02:13.192664+0000","last_unstale":"2024-04-25T14:02:14.620239+0000","last_undegraded":"2024-04-25T14:02:14.620239+0000","last_fullsized":"2024-04-25T14:02:14.620239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621118+0000","last_change":"2024-04-25T14:02:09.175890+0000","last_active":"2024-04-25T14:02:14.621118+0000","last_peered":"2024-04-25T14:02:14.621118+0000","last_clean":"2024-04-25T14:02:14.621118+0000","last_became_active":"2024-04-25T14:02:09.175586+0000","last_became_peered":"2024-04-25T14:02:09.175586+0000","last_unstale":"2024-04-25T14:02:14.621118+0000","last_undegraded":"2024-04-25T14:02:14.621118+0000","last_fullsized":"2024-04-25T14:02:14.621118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619271+0000","last_change":"2024-04-25T14:02:08.141597+0000","last_active":"2024-04-25T14:02:14.619271+0000","last_peered":"2024-04-25T14:02:14.619271+0000","last_clean":"2024-04-25T14:02:14.619271+0000","last_became_active":"2024-04-25T14:02:08.141271+0000","last_became_peered":"2024-04-25T14:02:08.141271+0000","last_unstale":"2024-04-25T14:02:14.619271+0000","last_undegraded":"2024-04-25T14:02:14.619271+0000","last_fullsized":"2024-04-25T14:02:14.619271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619661+0000","last_change":"2024-04-25T14:02:13.208359+0000","last_active":"2024-04-25T14:02:14.619661+0000","last_peered":"2024-04-25T14:02:14.619661+0000","last_clean":"2024-04-25T14:02:14.619661+0000","last_became_active":"2024-04-25T14:02:13.204049+0000","last_became_peered":"2024-04-25T14:02:13.204049+0000","last_unstale":"2024-04-25T14:02:14.619661+0000","last_undegraded":"2024-04-25T14:02:14.619661+0000","last_fullsized":"2024-04-25T14:02:14.619661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620199+0000","last_change":"2024-04-25T14:02:09.153508+0000","last_active":"2024-04-25T14:02:14.620199+0000","last_peered":"2024-04-25T14:02:14.620199+0000","last_clean":"2024-04-25T14:02:14.620199+0000","last_became_active":"2024-04-25T14:02:09.153087+0000","last_became_peered":"2024-04-25T14:02:09.153087+0000","last_unstale":"2024-04-25T14:02:14.620199+0000","last_undegraded":"2024-04-25T14:02:14.620199+0000","last_fullsized":"2024-04-25T14:02:14.620199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181562+0000","last_change":"2024-04-25T14:02:08.149363+0000","last_active":"2024-04-25T14:02:14.181562+0000","last_peered":"2024-04-25T14:02:14.181562+0000","last_clean":"2024-04-25T14:02:14.181562+0000","last_became_active":"2024-04-25T14:02:08.148936+0000","last_became_peered":"2024-04-25T14:02:08.148936+0000","last_unstale":"2024-04-25T14:02:14.181562+0000","last_undegraded":"2024-04-25T14:02:14.181562+0000","last_fullsized":"2024-04-25T14:02:14.181562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:33.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180089+0000","last_change":"2024-04-25T14:02:13.201594+0000","last_active":"2024-04-25T14:02:14.180089+0000","last_peered":"2024-04-25T14:02:14.180089+0000","last_clean":"2024-04-25T14:02:14.180089+0000","last_became_active":"2024-04-25T14:02:13.201156+0000","last_became_peered":"2024-04-25T14:02:13.201156+0000","last_unstale":"2024-04-25T14:02:14.180089+0000","last_undegraded":"2024-04-25T14:02:14.180089+0000","last_fullsized":"2024-04-25T14:02:14.180089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620504+0000","last_change":"2024-04-25T14:02:09.164648+0000","last_active":"2024-04-25T14:02:14.620504+0000","last_peered":"2024-04-25T14:02:14.620504+0000","last_clean":"2024-04-25T14:02:14.620504+0000","last_became_active":"2024-04-25T14:02:09.164319+0000","last_became_peered":"2024-04-25T14:02:09.164319+0000","last_unstale":"2024-04-25T14:02:14.620504+0000","last_undegraded":"2024-04-25T14:02:14.620504+0000","last_fullsized":"2024-04-25T14:02:14.620504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620863+0000","last_change":"2024-04-25T14:02:08.148046+0000","last_active":"2024-04-25T14:02:14.620863+0000","last_peered":"2024-04-25T14:02:14.620863+0000","last_clean":"2024-04-25T14:02:14.620863+0000","last_became_active":"2024-04-25T14:02:08.147707+0000","last_became_peered":"2024-04-25T14:02:08.147707+0000","last_unstale":"2024-04-25T14:02:14.620863+0000","last_undegraded":"2024-04-25T14:02:14.620863+0000","last_fullsized":"2024-04-25T14:02:14.620863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619751+0000","last_change":"2024-04-25T14:02:13.197572+0000","last_active":"2024-04-25T14:02:14.619751+0000","last_peered":"2024-04-25T14:02:14.619751+0000","last_clean":"2024-04-25T14:02:14.619751+0000","last_became_active":"2024-04-25T14:02:13.197138+0000","last_became_peered":"2024-04-25T14:02:13.197138+0000","last_unstale":"2024-04-25T14:02:14.619751+0000","last_undegraded":"2024-04-25T14:02:14.619751+0000","last_fullsized":"2024-04-25T14:02:14.619751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619861+0000","last_change":"2024-04-25T14:02:09.163807+0000","last_active":"2024-04-25T14:02:14.619861+0000","last_peered":"2024-04-25T14:02:14.619861+0000","last_clean":"2024-04-25T14:02:14.619861+0000","last_became_active":"2024-04-25T14:02:09.163510+0000","last_became_peered":"2024-04-25T14:02:09.163510+0000","last_unstale":"2024-04-25T14:02:14.619861+0000","last_undegraded":"2024-04-25T14:02:14.619861+0000","last_fullsized":"2024-04-25T14:02:14.619861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620015+0000","last_change":"2024-04-25T14:02:08.148684+0000","last_active":"2024-04-25T14:02:14.620015+0000","last_peered":"2024-04-25T14:02:14.620015+0000","last_clean":"2024-04-25T14:02:14.620015+0000","last_became_active":"2024-04-25T14:02:08.148384+0000","last_became_peered":"2024-04-25T14:02:08.148384+0000","last_unstale":"2024-04-25T14:02:14.620015+0000","last_undegraded":"2024-04-25T14:02:14.620015+0000","last_fullsized":"2024-04-25T14:02:14.620015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620837+0000","last_change":"2024-04-25T14:02:13.204849+0000","last_active":"2024-04-25T14:02:14.620837+0000","last_peered":"2024-04-25T14:02:14.620837+0000","last_clean":"2024-04-25T14:02:14.620837+0000","last_became_active":"2024-04-25T14:02:13.204459+0000","last_became_peered":"2024-04-25T14:02:13.204459+0000","last_unstale":"2024-04-25T14:02:14.620837+0000","last_undegraded":"2024-04-25T14:02:14.620837+0000","last_fullsized":"2024-04-25T14:02:14.620837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619156+0000","last_change":"2024-04-25T14:02:09.153461+0000","last_active":"2024-04-25T14:02:14.619156+0000","last_peered":"2024-04-25T14:02:14.619156+0000","last_clean":"2024-04-25T14:02:14.619156+0000","last_became_active":"2024-04-25T14:02:09.153122+0000","last_became_peered":"2024-04-25T14:02:09.153122+0000","last_unstale":"2024-04-25T14:02:14.619156+0000","last_undegraded":"2024-04-25T14:02:14.619156+0000","last_fullsized":"2024-04-25T14:02:14.619156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619253+0000","last_change":"2024-04-25T14:02:08.141488+0000","last_active":"2024-04-25T14:02:14.619253+0000","last_peered":"2024-04-25T14:02:14.619253+0000","last_clean":"2024-04-25T14:02:14.619253+0000","last_became_active":"2024-04-25T14:02:08.141195+0000","last_became_peered":"2024-04-25T14:02:08.141195+0000","last_unstale":"2024-04-25T14:02:14.619253+0000","last_undegraded":"2024-04-25T14:02:14.619253+0000","last_fullsized":"2024-04-25T14:02:14.619253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620173+0000","last_change":"2024-04-25T14:02:13.202926+0000","last_active":"2024-04-25T14:02:14.620173+0000","last_peered":"2024-04-25T14:02:14.620173+0000","last_clean":"2024-04-25T14:02:14.620173+0000","last_became_active":"2024-04-25T14:02:13.202452+0000","last_became_peered":"2024-04-25T14:02:13.202452+0000","last_unstale":"2024-04-25T14:02:14.620173+0000","last_undegraded":"2024-04-25T14:02:14.620173+0000","last_fullsized":"2024-04-25T14:02:14.620173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179693+0000","last_change":"2024-04-25T14:02:09.162133+0000","last_active":"2024-04-25T14:02:14.179693+0000","last_peered":"2024-04-25T14:02:14.179693+0000","last_clean":"2024-04-25T14:02:14.179693+0000","last_became_active":"2024-04-25T14:02:09.161795+0000","last_became_peered":"2024-04-25T14:02:09.161795+0000","last_unstale":"2024-04-25T14:02:14.179693+0000","last_undegraded":"2024-04-25T14:02:14.179693+0000","last_fullsized":"2024-04-25T14:02:14.179693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:49.968260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180083+0000","last_change":"2024-04-25T14:02:08.144670+0000","last_active":"2024-04-25T14:02:14.180083+0000","last_peered":"2024-04-25T14:02:14.180083+0000","last_clean":"2024-04-25T14:02:14.180083+0000","last_became_active":"2024-04-25T14:02:08.143303+0000","last_became_peered":"2024-04-25T14:02:08.143303+0000","last_unstale":"2024-04-25T14:02:14.180083+0000","last_undegraded":"2024-04-25T14:02:14.180083+0000","last_fullsized":"2024-04-25T14:02:14.180083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.199069+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.198521+0000","last_became_peered":"2024-04-25T14:02:13.198521+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.164561+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.164215+0000","last_became_peered":"2024-04-25T14:02:09.164215+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619618+0000","last_change":"2024-04-25T14:02:08.150157+0000","last_active":"2024-04-25T14:02:14.619618+0000","last_peered":"2024-04-25T14:02:14.619618+0000","last_clean":"2024-04-25T14:02:14.619618+0000","last_became_active":"2024-04-25T14:02:08.149812+0000","last_became_peered":"2024-04-25T14:02:08.149812+0000","last_unstale":"2024-04-25T14:02:14.619618+0000","last_undegraded":"2024-04-25T14:02:14.619618+0000","last_fullsized":"2024-04-25T14:02:14.619618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180256+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:02:14.180256+0000","last_peered":"2024-04-25T14:02:14.180256+0000","last_clean":"2024-04-25T14:02:14.180256+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:02:14.180256+0000","last_undegraded":"2024-04-25T14:02:14.180256+0000","last_fullsized":"2024-04-25T14:02:14.180256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.202985+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.202514+0000","last_became_peered":"2024-04-25T14:02:13.202514+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181284+0000","last_change":"2024-04-25T14:02:09.164697+0000","last_active":"2024-04-25T14:02:14.181284+0000","last_peered":"2024-04-25T14:02:14.181284+0000","last_clean":"2024-04-25T14:02:14.181284+0000","last_became_active":"2024-04-25T14:02:09.164352+0000","last_became_peered":"2024-04-25T14:02:09.164352+0000","last_unstale":"2024-04-25T14:02:14.181284+0000","last_undegraded":"2024-04-25T14:02:14.181284+0000","last_fullsized":"2024-04-25T14:02:14.181284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620427+0000","last_change":"2024-04-25T14:02:08.150437+0000","last_active":"2024-04-25T14:02:14.620427+0000","last_peered":"2024-04-25T14:02:14.620427+0000","last_clean":"2024-04-25T14:02:14.620427+0000","last_became_active":"2024-04-25T14:02:08.149954+0000","last_became_peered":"2024-04-25T14:02:08.149954+0000","last_unstale":"2024-04-25T14:02:14.620427+0000","last_undegraded":"2024-04-25T14:02:14.620427+0000","last_fullsized":"2024-04-25T14:02:14.620427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181863+0000","last_change":"2024-04-25T14:02:13.198918+0000","last_active":"2024-04-25T14:02:14.181863+0000","last_peered":"2024-04-25T14:02:14.181863+0000","last_clean":"2024-04-25T14:02:14.181863+0000","last_became_active":"2024-04-25T14:02:13.198244+0000","last_became_peered":"2024-04-25T14:02:13.198244+0000","last_unstale":"2024-04-25T14:02:14.181863+0000","last_undegraded":"2024-04-25T14:02:14.181863+0000","last_fullsized":"2024-04-25T14:02:14.181863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619706+0000","last_change":"2024-04-25T14:02:09.162899+0000","last_active":"2024-04-25T14:02:14.619706+0000","last_peered":"2024-04-25T14:02:14.619706+0000","last_clean":"2024-04-25T14:02:14.619706+0000","last_became_active":"2024-04-25T14:02:09.162573+0000","last_became_peered":"2024-04-25T14:02:09.162573+0000","last_unstale":"2024-04-25T14:02:14.619706+0000","last_undegraded":"2024-04-25T14:02:14.619706+0000","last_fullsized":"2024-04-25T14:02:14.619706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621027+0000","last_change":"2024-04-25T14:02:08.140157+0000","last_active":"2024-04-25T14:02:14.621027+0000","last_peered":"2024-04-25T14:02:14.621027+0000","last_clean":"2024-04-25T14:02:14.621027+0000","last_became_active":"2024-04-25T14:02:08.139477+0000","last_became_peered":"2024-04-25T14:02:08.139477+0000","last_unstale":"2024-04-25T14:02:14.621027+0000","last_undegraded":"2024-04-25T14:02:14.621027+0000","last_fullsized":"2024-04-25T14:02:14.621027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180180+0000","last_change":"2024-04-25T14:02:13.201803+0000","last_active":"2024-04-25T14:02:14.180180+0000","last_peered":"2024-04-25T14:02:14.180180+0000","last_clean":"2024-04-25T14:02:14.180180+0000","last_became_active":"2024-04-25T14:02:13.201388+0000","last_became_peered":"2024-04-25T14:02:13.201388+0000","last_unstale":"2024-04-25T14:02:14.180180+0000","last_undegraded":"2024-04-25T14:02:14.180180+0000","last_fullsized":"2024-04-25T14:02:14.180180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182344+0000","last_change":"2024-04-25T14:02:09.171757+0000","last_active":"2024-04-25T14:02:14.182344+0000","last_peered":"2024-04-25T14:02:14.182344+0000","last_clean":"2024-04-25T14:02:14.182344+0000","last_became_active":"2024-04-25T14:02:09.168908+0000","last_became_peered":"2024-04-25T14:02:09.168908+0000","last_unstale":"2024-04-25T14:02:14.182344+0000","last_undegraded":"2024-04-25T14:02:14.182344+0000","last_fullsized":"2024-04-25T14:02:14.182344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:21.339622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620710+0000","last_change":"2024-04-25T14:02:08.140886+0000","last_active":"2024-04-25T14:02:14.620710+0000","last_peered":"2024-04-25T14:02:14.620710+0000","last_clean":"2024-04-25T14:02:14.620710+0000","last_became_active":"2024-04-25T14:02:08.138585+0000","last_became_peered":"2024-04-25T14:02:08.138585+0000","last_unstale":"2024-04-25T14:02:14.620710+0000","last_undegraded":"2024-04-25T14:02:14.620710+0000","last_fullsized":"2024-04-25T14:02:14.620710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619005+0000","last_change":"2024-04-25T14:02:13.194033+0000","last_active":"2024-04-25T14:02:14.619005+0000","last_peered":"2024-04-25T14:02:14.619005+0000","last_clean":"2024-04-25T14:02:14.619005+0000","last_became_active":"2024-04-25T14:02:13.193486+0000","last_became_peered":"2024-04-25T14:02:13.193486+0000","last_unstale":"2024-04-25T14:02:14.619005+0000","last_undegraded":"2024-04-25T14:02:14.619005+0000","last_fullsized":"2024-04-25T14:02:14.619005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181242+0000","last_change":"2024-04-25T14:02:09.154176+0000","last_active":"2024-04-25T14:02:14.181242+0000","last_peered":"2024-04-25T14:02:14.181242+0000","last_clean":"2024-04-25T14:02:14.181242+0000","last_became_active":"2024-04-25T14:02:09.153867+0000","last_became_peered":"2024-04-25T14:02:09.153867+0000","last_unstale":"2024-04-25T14:02:14.181242+0000","last_undegraded":"2024-04-25T14:02:14.181242+0000","last_fullsized":"2024-04-25T14:02:14.181242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619639+0000","last_change":"2024-04-25T14:02:08.148162+0000","last_active":"2024-04-25T14:02:14.619639+0000","last_peered":"2024-04-25T14:02:14.619639+0000","last_clean":"2024-04-25T14:02:14.619639+0000","last_became_active":"2024-04-25T14:02:08.147822+0000","last_became_peered":"2024-04-25T14:02:08.147822+0000","last_unstale":"2024-04-25T14:02:14.619639+0000","last_undegraded":"2024-04-25T14:02:14.619639+0000","last_fullsized":"2024-04-25T14:02:14.619639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:47.001110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619644+0000","last_change":"2024-04-25T14:02:13.204464+0000","last_active":"2024-04-25T14:02:14.619644+0000","last_peered":"2024-04-25T14:02:14.619644+0000","last_clean":"2024-04-25T14:02:14.619644+0000","last_became_active":"2024-04-25T14:02:13.203501+0000","last_became_peered":"2024-04-25T14:02:13.203501+0000","last_unstale":"2024-04-25T14:02:14.619644+0000","last_undegraded":"2024-04-25T14:02:14.619644+0000","last_fullsized":"2024-04-25T14:02:14.619644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:36.450612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180471+0000","last_change":"2024-04-25T14:02:09.169427+0000","last_active":"2024-04-25T14:02:14.180471+0000","last_peered":"2024-04-25T14:02:14.180471+0000","last_clean":"2024-04-25T14:02:14.180471+0000","last_became_active":"2024-04-25T14:02:09.168920+0000","last_became_peered":"2024-04-25T14:02:09.168920+0000","last_unstale":"2024-04-25T14:02:14.180471+0000","last_undegraded":"2024-04-25T14:02:14.180471+0000","last_fullsized":"2024-04-25T14:02:14.180471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618690+0000","last_change":"2024-04-25T14:02:08.143121+0000","last_active":"2024-04-25T14:02:14.618690+0000","last_peered":"2024-04-25T14:02:14.618690+0000","last_clean":"2024-04-25T14:02:14.618690+0000","last_became_active":"2024-04-25T14:02:08.142765+0000","last_became_peered":"2024-04-25T14:02:08.142765+0000","last_unstale":"2024-04-25T14:02:14.618690+0000","last_undegraded":"2024-04-25T14:02:14.618690+0000","last_fullsized":"2024-04-25T14:02:14.618690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180517+0000","last_change":"2024-04-25T14:02:13.205101+0000","last_active":"2024-04-25T14:02:14.180517+0000","last_peered":"2024-04-25T14:02:14.180517+0000","last_clean":"2024-04-25T14:02:14.180517+0000","last_became_active":"2024-04-25T14:02:13.204696+0000","last_became_peered":"2024-04-25T14:02:13.204696+0000","last_unstale":"2024-04-25T14:02:14.180517+0000","last_undegraded":"2024-04-25T14:02:14.180517+0000","last_fullsized":"2024-04-25T14:02:14.180517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179895+0000","last_change":"2024-04-25T14:02:09.165439+0000","last_active":"2024-04-25T14:02:14.179895+0000","last_peered":"2024-04-25T14:02:14.179895+0000","last_clean":"2024-04-25T14:02:14.179895+0000","last_became_active":"2024-04-25T14:02:09.164934+0000","last_became_peered":"2024-04-25T14:02:09.164934+0000","last_unstale":"2024-04-25T14:02:14.179895+0000","last_undegraded":"2024-04-25T14:02:14.179895+0000","last_fullsized":"2024-04-25T14:02:14.179895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180810+0000","last_change":"2024-04-25T14:02:08.144013+0000","last_active":"2024-04-25T14:02:14.180810+0000","last_peered":"2024-04-25T14:02:14.180810+0000","last_clean":"2024-04-25T14:02:14.180810+0000","last_became_active":"2024-04-25T14:02:08.141723+0000","last_became_peered":"2024-04-25T14:02:08.141723+0000","last_unstale":"2024-04-25T14:02:14.180810+0000","last_undegraded":"2024-04-25T14:02:14.180810+0000","last_fullsized":"2024-04-25T14:02:14.180810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620974+0000","last_change":"2024-04-25T14:02:13.199511+0000","last_active":"2024-04-25T14:02:14.620974+0000","last_peered":"2024-04-25T14:02:14.620974+0000","last_clean":"2024-04-25T14:02:14.620974+0000","last_became_active":"2024-04-25T14:02:13.199074+0000","last_became_peered":"2024-04-25T14:02:13.199074+0000","last_unstale":"2024-04-25T14:02:14.620974+0000","last_undegraded":"2024-04-25T14:02:14.620974+0000","last_fullsized":"2024-04-25T14:02:14.620974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:47.660769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180811+0000","last_change":"2024-04-25T14:02:09.171097+0000","last_active":"2024-04-25T14:02:14.180811+0000","last_peered":"2024-04-25T14:02:14.180811+0000","last_clean":"2024-04-25T14:02:14.180811+0000","last_became_active":"2024-04-25T14:02:09.169656+0000","last_became_peered":"2024-04-25T14:02:09.169656+0000","last_unstale":"2024-04-25T14:02:14.180811+0000","last_undegraded":"2024-04-25T14:02:14.180811+0000","last_fullsized":"2024-04-25T14:02:14.180811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:34.914414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619583+0000","last_change":"2024-04-25T14:02:08.148427+0000","last_active":"2024-04-25T14:02:14.619583+0000","last_peered":"2024-04-25T14:02:14.619583+0000","last_clean":"2024-04-25T14:02:14.619583+0000","last_became_active":"2024-04-25T14:02:08.148019+0000","last_became_peered":"2024-04-25T14:02:08.148019+0000","last_unstale":"2024-04-25T14:02:14.619583+0000","last_undegraded":"2024-04-25T14:02:14.619583+0000","last_fullsized":"2024-04-25T14:02:14.619583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621238+0000","last_change":"2024-04-25T14:02:13.203865+0000","last_active":"2024-04-25T14:02:14.621238+0000","last_peered":"2024-04-25T14:02:14.621238+0000","last_clean":"2024-04-25T14:02:14.621238+0000","last_became_active":"2024-04-25T14:02:13.203056+0000","last_became_peered":"2024-04-25T14:02:13.203056+0000","last_unstale":"2024-04-25T14:02:14.621238+0000","last_undegraded":"2024-04-25T14:02:14.621238+0000","last_fullsized":"2024-04-25T14:02:14.621238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:58.555213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179953+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.179953+0000","last_peered":"2024-04-25T14:02:14.179953+0000","last_clean":"2024-04-25T14:02:14.179953+0000","last_became_active":"2024-04-25T14:02:09.171025+0000","last_became_peered":"2024-04-25T14:02:09.171025+0000","last_unstale":"2024-04-25T14:02:14.179953+0000","last_undegraded":"2024-04-25T14:02:14.179953+0000","last_fullsized":"2024-04-25T14:02:14.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180403+0000","last_change":"2024-04-25T14:02:08.141709+0000","last_active":"2024-04-25T14:02:14.180403+0000","last_peered":"2024-04-25T14:02:14.180403+0000","last_clean":"2024-04-25T14:02:14.180403+0000","last_became_active":"2024-04-25T14:02:08.141389+0000","last_became_peered":"2024-04-25T14:02:08.141389+0000","last_unstale":"2024-04-25T14:02:14.180403+0000","last_undegraded":"2024-04-25T14:02:14.180403+0000","last_fullsized":"2024-04-25T14:02:14.180403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620299+0000","last_change":"2024-04-25T14:02:13.205138+0000","last_active":"2024-04-25T14:02:14.620299+0000","last_peered":"2024-04-25T14:02:14.620299+0000","last_clean":"2024-04-25T14:02:14.620299+0000","last_became_active":"2024-04-25T14:02:13.204712+0000","last_became_peered":"2024-04-25T14:02:13.204712+0000","last_unstale":"2024-04-25T14:02:14.620299+0000","last_undegraded":"2024-04-25T14:02:14.620299+0000","last_fullsized":"2024-04-25T14:02:14.620299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181162+0000","last_change":"2024-04-25T14:02:09.159739+0000","last_active":"2024-04-25T14:02:14.181162+0000","last_peered":"2024-04-25T14:02:14.181162+0000","last_clean":"2024-04-25T14:02:14.181162+0000","last_became_active":"2024-04-25T14:02:09.158156+0000","last_became_peered":"2024-04-25T14:02:09.158156+0000","last_unstale":"2024-04-25T14:02:14.181162+0000","last_undegraded":"2024-04-25T14:02:14.181162+0000","last_fullsized":"2024-04-25T14:02:14.181162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:18.690850+0000","last_change":"2024-04-25T14:02:08.143908+0000","last_active":"2024-04-25T14:02:18.690850+0000","last_peered":"2024-04-25T14:02:18.690850+0000","last_clean":"2024-04-25T14:02:18.690850+0000","last_became_active":"2024-04-25T14:02:08.143478+0000","last_became_peered":"2024-04-25T14:02:08.143478+0000","last_unstale":"2024-04-25T14:02:18.690850+0000","last_undegraded":"2024-04-25T14:02:18.690850+0000","last_fullsized":"2024-04-25T14:02:18.690850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619926+0000","last_change":"2024-04-25T14:02:13.197367+0000","last_active":"2024-04-25T14:02:14.619926+0000","last_peered":"2024-04-25T14:02:14.619926+0000","last_clean":"2024-04-25T14:02:14.619926+0000","last_became_active":"2024-04-25T14:02:13.197029+0000","last_became_peered":"2024-04-25T14:02:13.197029+0000","last_unstale":"2024-04-25T14:02:14.619926+0000","last_undegraded":"2024-04-25T14:02:14.619926+0000","last_fullsized":"2024-04-25T14:02:14.619926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620116+0000","last_change":"2024-04-25T14:02:09.162706+0000","last_active":"2024-04-25T14:02:14.620116+0000","last_peered":"2024-04-25T14:02:14.620116+0000","last_clean":"2024-04-25T14:02:14.620116+0000","last_became_active":"2024-04-25T14:02:09.162316+0000","last_became_peered":"2024-04-25T14:02:09.162316+0000","last_unstale":"2024-04-25T14:02:14.620116+0000","last_undegraded":"2024-04-25T14:02:14.620116+0000","last_fullsized":"2024-04-25T14:02:14.620116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:23.993603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620975+0000","last_change":"2024-04-25T14:02:08.150406+0000","last_active":"2024-04-25T14:02:14.620975+0000","last_peered":"2024-04-25T14:02:14.620975+0000","last_clean":"2024-04-25T14:02:14.620975+0000","last_became_active":"2024-04-25T14:02:08.150057+0000","last_became_peered":"2024-04-25T14:02:08.150057+0000","last_unstale":"2024-04-25T14:02:14.620975+0000","last_undegraded":"2024-04-25T14:02:14.620975+0000","last_fullsized":"2024-04-25T14:02:14.620975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619602+0000","last_change":"2024-04-25T14:02:13.197851+0000","last_active":"2024-04-25T14:02:14.619602+0000","last_peered":"2024-04-25T14:02:14.619602+0000","last_clean":"2024-04-25T14:02:14.619602+0000","last_became_active":"2024-04-25T14:02:13.197109+0000","last_became_peered":"2024-04-25T14:02:13.197109+0000","last_unstale":"2024-04-25T14:02:14.619602+0000","last_undegraded":"2024-04-25T14:02:14.619602+0000","last_fullsized":"2024-04-25T14:02:14.619602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:20.798044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180425+0000","last_change":"2024-04-25T14:02:09.168810+0000","last_active":"2024-04-25T14:02:14.180425+0000","last_peered":"2024-04-25T14:02:14.180425+0000","last_clean":"2024-04-25T14:02:14.180425+0000","last_became_active":"2024-04-25T14:02:09.168323+0000","last_became_peered":"2024-04-25T14:02:09.168323+0000","last_unstale":"2024-04-25T14:02:14.180425+0000","last_undegraded":"2024-04-25T14:02:14.180425+0000","last_fullsized":"2024-04-25T14:02:14.180425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619958+0000","last_change":"2024-04-25T14:02:08.145115+0000","last_active":"2024-04-25T14:02:14.619958+0000","last_peered":"2024-04-25T14:02:14.619958+0000","last_clean":"2024-04-25T14:02:14.619958+0000","last_became_active":"2024-04-25T14:02:08.144853+0000","last_became_peered":"2024-04-25T14:02:08.144853+0000","last_unstale":"2024-04-25T14:02:14.619958+0000","last_undegraded":"2024-04-25T14:02:14.619958+0000","last_fullsized":"2024-04-25T14:02:14.619958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619831+0000","last_change":"2024-04-25T14:02:13.203422+0000","last_active":"2024-04-25T14:02:14.619831+0000","last_peered":"2024-04-25T14:02:14.619831+0000","last_clean":"2024-04-25T14:02:14.619831+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619831+0000","last_undegraded":"2024-04-25T14:02:14.619831+0000","last_fullsized":"2024-04-25T14:02:14.619831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619921+0000","last_change":"2024-04-25T14:02:09.171494+0000","last_active":"2024-04-25T14:02:14.619921+0000","last_peered":"2024-04-25T14:02:14.619921+0000","last_clean":"2024-04-25T14:02:14.619921+0000","last_became_active":"2024-04-25T14:02:09.171163+0000","last_became_peered":"2024-04-25T14:02:09.171163+0000","last_unstale":"2024-04-25T14:02:14.619921+0000","last_undegraded":"2024-04-25T14:02:14.619921+0000","last_fullsized":"2024-04-25T14:02:14.619921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181248+0000","last_change":"2024-04-25T14:02:08.150536+0000","last_active":"2024-04-25T14:02:14.181248+0000","last_peered":"2024-04-25T14:02:14.181248+0000","last_clean":"2024-04-25T14:02:14.181248+0000","last_became_active":"2024-04-25T14:02:08.150156+0000","last_became_peered":"2024-04-25T14:02:08.150156+0000","last_unstale":"2024-04-25T14:02:14.181248+0000","last_undegraded":"2024-04-25T14:02:14.181248+0000","last_fullsized":"2024-04-25T14:02:14.181248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619292+0000","last_change":"2024-04-25T14:02:13.201000+0000","last_active":"2024-04-25T14:02:14.619292+0000","last_peered":"2024-04-25T14:02:14.619292+0000","last_clean":"2024-04-25T14:02:14.619292+0000","last_became_active":"2024-04-25T14:02:13.200594+0000","last_became_peered":"2024-04-25T14:02:13.200594+0000","last_unstale":"2024-04-25T14:02:14.619292+0000","last_undegraded":"2024-04-25T14:02:14.619292+0000","last_fullsized":"2024-04-25T14:02:14.619292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.706373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180405+0000","last_change":"2024-04-25T14:02:13.204286+0000","last_active":"2024-04-25T14:02:14.180405+0000","last_peered":"2024-04-25T14:02:14.180405+0000","last_clean":"2024-04-25T14:02:14.180405+0000","last_became_active":"2024-04-25T14:02:13.203885+0000","last_became_peered":"2024-04-25T14:02:13.203885+0000","last_unstale":"2024-04-25T14:02:14.180405+0000","last_undegraded":"2024-04-25T14:02:14.180405+0000","last_fullsized":"2024-04-25T14:02:14.180405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619363+0000","last_change":"2024-04-25T14:02:09.158647+0000","last_active":"2024-04-25T14:02:14.619363+0000","last_peered":"2024-04-25T14:02:14.619363+0000","last_clean":"2024-04-25T14:02:14.619363+0000","last_became_active":"2024-04-25T14:02:09.158307+0000","last_became_peered":"2024-04-25T14:02:09.158307+0000","last_unstale":"2024-04-25T14:02:14.619363+0000","last_undegraded":"2024-04-25T14:02:14.619363+0000","last_fullsized":"2024-04-25T14:02:14.619363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619239+0000","last_change":"2024-04-25T14:02:08.149002+0000","last_active":"2024-04-25T14:02:14.619239+0000","last_peered":"2024-04-25T14:02:14.619239+0000","last_clean":"2024-04-25T14:02:14.619239+0000","last_became_active":"2024-04-25T14:02:08.148700+0000","last_became_peered":"2024-04-25T14:02:08.148700+0000","last_unstale":"2024-04-25T14:02:14.619239+0000","last_undegraded":"2024-04-25T14:02:14.619239+0000","last_fullsized":"2024-04-25T14:02:14.619239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181096+0000","last_change":"2024-04-25T14:02:13.197649+0000","last_active":"2024-04-25T14:02:14.181096+0000","last_peered":"2024-04-25T14:02:14.181096+0000","last_clean":"2024-04-25T14:02:14.181096+0000","last_became_active":"2024-04-25T14:02:13.197203+0000","last_became_peered":"2024-04-25T14:02:13.197203+0000","last_unstale":"2024-04-25T14:02:14.181096+0000","last_undegraded":"2024-04-25T14:02:14.181096+0000","last_fullsized":"2024-04-25T14:02:14.181096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619786+0000","last_change":"2024-04-25T14:02:09.164726+0000","last_active":"2024-04-25T14:02:14.619786+0000","last_peered":"2024-04-25T14:02:14.619786+0000","last_clean":"2024-04-25T14:02:14.619786+0000","last_became_active":"2024-04-25T14:02:09.164047+0000","last_became_peered":"2024-04-25T14:02:09.164047+0000","last_unstale":"2024-04-25T14:02:14.619786+0000","last_undegraded":"2024-04-25T14:02:14.619786+0000","last_fullsized":"2024-04-25T14:02:14.619786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619586+0000","last_change":"2024-04-25T14:02:08.138596+0000","last_active":"2024-04-25T14:02:14.619586+0000","last_peered":"2024-04-25T14:02:14.619586+0000","last_clean":"2024-04-25T14:02:14.619586+0000","last_became_active":"2024-04-25T14:02:08.138268+0000","last_became_peered":"2024-04-25T14:02:08.138268+0000","last_unstale":"2024-04-25T14:02:14.619586+0000","last_undegraded":"2024-04-25T14:02:14.619586+0000","last_fullsized":"2024-04-25T14:02:14.619586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182061+0000","last_change":"2024-04-25T14:02:08.144180+0000","last_active":"2024-04-25T14:02:14.182061+0000","last_peered":"2024-04-25T14:02:14.182061+0000","last_clean":"2024-04-25T14:02:14.182061+0000","last_became_active":"2024-04-25T14:02:08.142554+0000","last_became_peered":"2024-04-25T14:02:08.142554+0000","last_unstale":"2024-04-25T14:02:14.182061+0000","last_undegraded":"2024-04-25T14:02:14.182061+0000","last_fullsized":"2024-04-25T14:02:14.182061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.163851+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.163561+0000","last_became_peered":"2024-04-25T14:02:09.163561+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620647+0000","last_change":"2024-04-25T14:02:13.200670+0000","last_active":"2024-04-25T14:02:14.620647+0000","last_peered":"2024-04-25T14:02:14.620647+0000","last_clean":"2024-04-25T14:02:14.620647+0000","last_became_active":"2024-04-25T14:02:13.200255+0000","last_became_peered":"2024-04-25T14:02:13.200255+0000","last_unstale":"2024-04-25T14:02:14.620647+0000","last_undegraded":"2024-04-25T14:02:14.620647+0000","last_fullsized":"2024-04-25T14:02:14.620647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:49.451815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620669+0000","last_change":"2024-04-25T14:02:08.136214+0000","last_active":"2024-04-25T14:02:14.620669+0000","last_peered":"2024-04-25T14:02:14.620669+0000","last_clean":"2024-04-25T14:02:14.620669+0000","last_became_active":"2024-04-25T14:02:08.135892+0000","last_became_peered":"2024-04-25T14:02:08.135892+0000","last_unstale":"2024-04-25T14:02:14.620669+0000","last_undegraded":"2024-04-25T14:02:14.620669+0000","last_fullsized":"2024-04-25T14:02:14.620669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:00.331661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.160067+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.159705+0000","last_became_peered":"2024-04-25T14:02:09.159705+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:21.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621148+0000","last_change":"2024-04-25T14:02:13.203327+0000","last_active":"2024-04-25T14:02:14.621148+0000","last_peered":"2024-04-25T14:02:14.621148+0000","last_clean":"2024-04-25T14:02:14.621148+0000","last_became_active":"2024-04-25T14:02:13.202870+0000","last_became_peered":"2024-04-25T14:02:13.202870+0000","last_unstale":"2024-04-25T14:02:14.621148+0000","last_undegraded":"2024-04-25T14:02:14.621148+0000","last_fullsized":"2024-04-25T14:02:14.621148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:39.636544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179770+0000","last_change":"2024-04-25T14:02:09.165255+0000","last_active":"2024-04-25T14:02:14.179770+0000","last_peered":"2024-04-25T14:02:14.179770+0000","last_clean":"2024-04-25T14:02:14.179770+0000","last_became_active":"2024-04-25T14:02:09.164886+0000","last_became_peered":"2024-04-25T14:02:09.164886+0000","last_unstale":"2024-04-25T14:02:14.179770+0000","last_undegraded":"2024-04-25T14:02:14.179770+0000","last_fullsized":"2024-04-25T14:02:14.179770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619161+0000","last_change":"2024-04-25T14:02:09.161880+0000","last_active":"2024-04-25T14:02:14.619161+0000","last_peered":"2024-04-25T14:02:14.619161+0000","last_clean":"2024-04-25T14:02:14.619161+0000","last_became_active":"2024-04-25T14:02:09.161568+0000","last_became_peered":"2024-04-25T14:02:09.161568+0000","last_unstale":"2024-04-25T14:02:14.619161+0000","last_undegraded":"2024-04-25T14:02:14.619161+0000","last_fullsized":"2024-04-25T14:02:14.619161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619443+0000","last_change":"2024-04-25T14:02:09.163237+0000","last_active":"2024-04-25T14:02:14.619443+0000","last_peered":"2024-04-25T14:02:14.619443+0000","last_clean":"2024-04-25T14:02:14.619443+0000","last_became_active":"2024-04-25T14:02:09.162870+0000","last_became_peered":"2024-04-25T14:02:09.162870+0000","last_unstale":"2024-04-25T14:02:14.619443+0000","last_undegraded":"2024-04-25T14:02:14.619443+0000","last_fullsized":"2024-04-25T14:02:14.619443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180151+0000","last_change":"2024-04-25T14:02:09.167689+0000","last_active":"2024-04-25T14:02:14.180151+0000","last_peered":"2024-04-25T14:02:14.180151+0000","last_clean":"2024-04-25T14:02:14.180151+0000","last_became_active":"2024-04-25T14:02:09.167379+0000","last_became_peered":"2024-04-25T14:02:09.167379+0000","last_unstale":"2024-04-25T14:02:14.180151+0000","last_undegraded":"2024-04-25T14:02:14.180151+0000","last_fullsized":"2024-04-25T14:02:14.180151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621304+0000","last_change":"2024-04-25T14:02:09.155156+0000","last_active":"2024-04-25T14:02:14.621304+0000","last_peered":"2024-04-25T14:02:14.621304+0000","last_clean":"2024-04-25T14:02:14.621304+0000","last_became_active":"2024-04-25T14:02:09.154497+0000","last_became_peered":"2024-04-25T14:02:09.154497+0000","last_unstale":"2024-04-25T14:02:14.621304+0000","last_undegraded":"2024-04-25T14:02:14.621304+0000","last_fullsized":"2024-04-25T14:02:14.621304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620178+0000","last_change":"2024-04-25T14:02:09.161627+0000","last_active":"2024-04-25T14:02:14.620178+0000","last_peered":"2024-04-25T14:02:14.620178+0000","last_clean":"2024-04-25T14:02:14.620178+0000","last_became_active":"2024-04-25T14:02:09.161321+0000","last_became_peered":"2024-04-25T14:02:09.161321+0000","last_unstale":"2024-04-25T14:02:14.620178+0000","last_undegraded":"2024-04-25T14:02:14.620178+0000","last_fullsized":"2024-04-25T14:02:14.620178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181400+0000","last_change":"2024-04-25T14:02:09.177833+0000","last_active":"2024-04-25T14:02:14.181400+0000","last_peered":"2024-04-25T14:02:14.181400+0000","last_clean":"2024-04-25T14:02:14.181400+0000","last_became_active":"2024-04-25T14:02:09.177530+0000","last_became_peered":"2024-04-25T14:02:09.177530+0000","last_unstale":"2024-04-25T14:02:14.181400+0000","last_undegraded":"2024-04-25T14:02:14.181400+0000","last_fullsized":"2024-04-25T14:02:14.181400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621026+0000","last_change":"2024-04-25T14:02:09.160341+0000","last_active":"2024-04-25T14:02:14.621026+0000","last_peered":"2024-04-25T14:02:14.621026+0000","last_clean":"2024-04-25T14:02:14.621026+0000","last_became_active":"2024-04-25T14:02:09.160028+0000","last_became_peered":"2024-04-25T14:02:09.160028+0000","last_unstale":"2024-04-25T14:02:14.621026+0000","last_undegraded":"2024-04-25T14:02:14.621026+0000","last_fullsized":"2024-04-25T14:02:14.621026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619081+0000","last_change":"2024-04-25T14:02:09.163936+0000","last_active":"2024-04-25T14:02:14.619081+0000","last_peered":"2024-04-25T14:02:14.619081+0000","last_clean":"2024-04-25T14:02:14.619081+0000","last_became_active":"2024-04-25T14:02:09.163303+0000","last_became_peered":"2024-04-25T14:02:09.163303+0000","last_unstale":"2024-04-25T14:02:14.619081+0000","last_undegraded":"2024-04-25T14:02:14.619081+0000","last_fullsized":"2024-04-25T14:02:14.619081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619413+0000","last_change":"2024-04-25T14:02:09.154839+0000","last_active":"2024-04-25T14:02:14.619413+0000","last_peered":"2024-04-25T14:02:14.619413+0000","last_clean":"2024-04-25T14:02:14.619413+0000","last_became_active":"2024-04-25T14:02:09.154443+0000","last_became_peered":"2024-04-25T14:02:09.154443+0000","last_unstale":"2024-04-25T14:02:14.619413+0000","last_undegraded":"2024-04-25T14:02:14.619413+0000","last_fullsized":"2024-04-25T14:02:14.619413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619286+0000","last_change":"2024-04-25T14:02:09.171782+0000","last_active":"2024-04-25T14:02:14.619286+0000","last_peered":"2024-04-25T14:02:14.619286+0000","last_clean":"2024-04-25T14:02:14.619286+0000","last_became_active":"2024-04-25T14:02:09.171158+0000","last_became_peered":"2024-04-25T14:02:09.171158+0000","last_unstale":"2024-04-25T14:02:14.619286+0000","last_undegraded":"2024-04-25T14:02:14.619286+0000","last_fullsized":"2024-04-25T14:02:14.619286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179956+0000","last_change":"2024-04-25T14:02:09.177793+0000","last_active":"2024-04-25T14:02:14.179956+0000","last_peered":"2024-04-25T14:02:14.179956+0000","last_clean":"2024-04-25T14:02:14.179956+0000","last_became_active":"2024-04-25T14:02:09.177475+0000","last_became_peered":"2024-04-25T14:02:09.177475+0000","last_unstale":"2024-04-25T14:02:14.179956+0000","last_undegraded":"2024-04-25T14:02:14.179956+0000","last_fullsized":"2024-04-25T14:02:14.179956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181451+0000","last_change":"2024-04-25T14:02:09.166782+0000","last_active":"2024-04-25T14:02:14.181451+0000","last_peered":"2024-04-25T14:02:14.181451+0000","last_clean":"2024-04-25T14:02:14.181451+0000","last_became_active":"2024-04-25T14:02:09.166277+0000","last_became_peered":"2024-04-25T14:02:09.166277+0000","last_unstale":"2024-04-25T14:02:14.181451+0000","last_undegraded":"2024-04-25T14:02:14.181451+0000","last_fullsized":"2024-04-25T14:02:14.181451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181472+0000","last_change":"2024-04-25T14:02:09.167448+0000","last_active":"2024-04-25T14:02:14.181472+0000","last_peered":"2024-04-25T14:02:14.181472+0000","last_clean":"2024-04-25T14:02:14.181472+0000","last_became_active":"2024-04-25T14:02:09.167136+0000","last_became_peered":"2024-04-25T14:02:09.167136+0000","last_unstale":"2024-04-25T14:02:14.181472+0000","last_undegraded":"2024-04-25T14:02:14.181472+0000","last_fullsized":"2024-04-25T14:02:14.181472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620713+0000","last_change":"2024-04-25T14:02:09.167572+0000","last_active":"2024-04-25T14:02:14.620713+0000","last_peered":"2024-04-25T14:02:14.620713+0000","last_clean":"2024-04-25T14:02:14.620713+0000","last_became_active":"2024-04-25T14:02:09.167227+0000","last_became_peered":"2024-04-25T14:02:09.167227+0000","last_unstale":"2024-04-25T14:02:14.620713+0000","last_undegraded":"2024-04-25T14:02:14.620713+0000","last_fullsized":"2024-04-25T14:02:14.620713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620673+0000","last_change":"2024-04-25T14:02:09.174321+0000","last_active":"2024-04-25T14:02:14.620673+0000","last_peered":"2024-04-25T14:02:14.620673+0000","last_clean":"2024-04-25T14:02:14.620673+0000","last_became_active":"2024-04-25T14:02:09.174015+0000","last_became_peered":"2024-04-25T14:02:09.174015+0000","last_unstale":"2024-04-25T14:02:14.620673+0000","last_undegraded":"2024-04-25T14:02:14.620673+0000","last_fullsized":"2024-04-25T14:02:14.620673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618957+0000","last_change":"2024-04-25T14:02:09.169554+0000","last_active":"2024-04-25T14:02:14.618957+0000","last_peered":"2024-04-25T14:02:14.618957+0000","last_clean":"2024-04-25T14:02:14.618957+0000","last_became_active":"2024-04-25T14:02:09.168555+0000","last_became_peered":"2024-04-25T14:02:09.168555+0000","last_unstale":"2024-04-25T14:02:14.618957+0000","last_undegraded":"2024-04-25T14:02:14.618957+0000","last_fullsized":"2024-04-25T14:02:14.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619417+0000","last_change":"2024-04-25T14:02:09.160170+0000","last_active":"2024-04-25T14:02:14.619417+0000","last_peered":"2024-04-25T14:02:14.619417+0000","last_clean":"2024-04-25T14:02:14.619417+0000","last_became_active":"2024-04-25T14:02:09.159871+0000","last_became_peered":"2024-04-25T14:02:09.159871+0000","last_unstale":"2024-04-25T14:02:14.619417+0000","last_undegraded":"2024-04-25T14:02:14.619417+0000","last_fullsized":"2024-04-25T14:02:14.619417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179980+0000","last_change":"2024-04-25T14:02:09.159887+0000","last_active":"2024-04-25T14:02:14.179980+0000","last_peered":"2024-04-25T14:02:14.179980+0000","last_clean":"2024-04-25T14:02:14.179980+0000","last_became_active":"2024-04-25T14:02:09.159431+0000","last_became_peered":"2024-04-25T14:02:09.159431+0000","last_unstale":"2024-04-25T14:02:14.179980+0000","last_undegraded":"2024-04-25T14:02:14.179980+0000","last_fullsized":"2024-04-25T14:02:14.179980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620557+0000","last_change":"2024-04-25T14:02:09.167290+0000","last_active":"2024-04-25T14:02:14.620557+0000","last_peered":"2024-04-25T14:02:14.620557+0000","last_clean":"2024-04-25T14:02:14.620557+0000","last_became_active":"2024-04-25T14:02:09.166974+0000","last_became_peered":"2024-04-25T14:02:09.166974+0000","last_unstale":"2024-04-25T14:02:14.620557+0000","last_undegraded":"2024-04-25T14:02:14.620557+0000","last_fullsized":"2024-04-25T14:02:14.620557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620085+0000","last_change":"2024-04-25T14:02:09.169185+0000","last_active":"2024-04-25T14:02:14.620085+0000","last_peered":"2024-04-25T14:02:14.620085+0000","last_clean":"2024-04-25T14:02:14.620085+0000","last_became_active":"2024-04-25T14:02:09.168862+0000","last_became_peered":"2024-04-25T14:02:09.168862+0000","last_unstale":"2024-04-25T14:02:14.620085+0000","last_undegraded":"2024-04-25T14:02:14.620085+0000","last_fullsized":"2024-04-25T14:02:14.620085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180614+0000","last_change":"2024-04-25T14:02:09.174006+0000","last_active":"2024-04-25T14:02:14.180614+0000","last_peered":"2024-04-25T14:02:14.180614+0000","last_clean":"2024-04-25T14:02:14.180614+0000","last_became_active":"2024-04-25T14:02:09.173662+0000","last_became_peered":"2024-04-25T14:02:09.173662+0000","last_unstale":"2024-04-25T14:02:14.180614+0000","last_undegraded":"2024-04-25T14:02:14.180614+0000","last_fullsized":"2024-04-25T14:02:14.180614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181263+0000","last_change":"2024-04-25T14:02:09.174331+0000","last_active":"2024-04-25T14:02:14.181263+0000","last_peered":"2024-04-25T14:02:14.181263+0000","last_clean":"2024-04-25T14:02:14.181263+0000","last_became_active":"2024-04-25T14:02:09.173998+0000","last_became_peered":"2024-04-25T14:02:09.173998+0000","last_unstale":"2024-04-25T14:02:14.181263+0000","last_undegraded":"2024-04-25T14:02:14.181263+0000","last_fullsized":"2024-04-25T14:02:14.181263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182263+0000","last_change":"2024-04-25T14:02:09.166767+0000","last_active":"2024-04-25T14:02:14.182263+0000","last_peered":"2024-04-25T14:02:14.182263+0000","last_clean":"2024-04-25T14:02:14.182263+0000","last_became_active":"2024-04-25T14:02:09.165131+0000","last_became_peered":"2024-04-25T14:02:09.165131+0000","last_unstale":"2024-04-25T14:02:14.182263+0000","last_undegraded":"2024-04-25T14:02:14.182263+0000","last_fullsized":"2024-04-25T14:02:14.182263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:07.386855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619681+0000","last_change":"2024-04-25T14:02:09.164960+0000","last_active":"2024-04-25T14:02:14.619681+0000","last_peered":"2024-04-25T14:02:14.619681+0000","last_clean":"2024-04-25T14:02:14.619681+0000","last_became_active":"2024-04-25T14:02:09.164417+0000","last_became_peered":"2024-04-25T14:02:09.164417+0000","last_unstale":"2024-04-25T14:02:14.619681+0000","last_undegraded":"2024-04-25T14:02:14.619681+0000","last_fullsized":"2024-04-25T14:02:14.619681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180423+0000","last_change":"2024-04-25T14:02:09.161177+0000","last_active":"2024-04-25T14:02:14.180423+0000","last_peered":"2024-04-25T14:02:14.180423+0000","last_clean":"2024-04-25T14:02:14.180423+0000","last_became_active":"2024-04-25T14:02:09.160404+0000","last_became_peered":"2024-04-25T14:02:09.160404+0000","last_unstale":"2024-04-25T14:02:14.180423+0000","last_undegraded":"2024-04-25T14:02:14.180423+0000","last_fullsized":"2024-04-25T14:02:14.180423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619241+0000","last_change":"2024-04-25T14:02:09.170498+0000","last_active":"2024-04-25T14:02:14.619241+0000","last_peered":"2024-04-25T14:02:14.619241+0000","last_clean":"2024-04-25T14:02:14.619241+0000","last_became_active":"2024-04-25T14:02:09.167994+0000","last_became_peered":"2024-04-25T14:02:09.167994+0000","last_unstale":"2024-04-25T14:02:14.619241+0000","last_undegraded":"2024-04-25T14:02:14.619241+0000","last_fullsized":"2024-04-25T14:02:14.619241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180565+0000","last_change":"2024-04-25T14:02:09.155470+0000","last_active":"2024-04-25T14:02:14.180565+0000","last_peered":"2024-04-25T14:02:14.180565+0000","last_clean":"2024-04-25T14:02:14.180565+0000","last_became_active":"2024-04-25T14:02:09.155123+0000","last_became_peered":"2024-04-25T14:02:09.155123+0000","last_unstale":"2024-04-25T14:02:14.180565+0000","last_undegraded":"2024-04-25T14:02:14.180565+0000","last_fullsized":"2024-04-25T14:02:14.180565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619577+0000","last_change":"2024-04-25T14:02:09.163082+0000","last_active":"2024-04-25T14:02:14.619577+0000","last_peered":"2024-04-25T14:02:14.619577+0000","last_clean":"2024-04-25T14:02:14.619577+0000","last_became_active":"2024-04-25T14:02:09.162148+0000","last_became_peered":"2024-04-25T14:02:09.162148+0000","last_unstale":"2024-04-25T14:02:14.619577+0000","last_undegraded":"2024-04-25T14:02:14.619577+0000","last_fullsized":"2024-04-25T14:02:14.619577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.167223+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.166924+0000","last_became_peered":"2024-04-25T14:02:09.166924+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619169+0000","last_change":"2024-04-25T14:02:09.159838+0000","last_active":"2024-04-25T14:02:14.619169+0000","last_peered":"2024-04-25T14:02:14.619169+0000","last_clean":"2024-04-25T14:02:14.619169+0000","last_became_active":"2024-04-25T14:02:09.159551+0000","last_became_peered":"2024-04-25T14:02:09.159551+0000","last_unstale":"2024-04-25T14:02:14.619169+0000","last_undegraded":"2024-04-25T14:02:14.619169+0000","last_fullsized":"2024-04-25T14:02:14.619169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182085+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.182085+0000","last_peered":"2024-04-25T14:02:14.182085+0000","last_clean":"2024-04-25T14:02:14.182085+0000","last_became_active":"2024-04-25T14:02:09.168711+0000","last_became_peered":"2024-04-25T14:02:09.168711+0000","last_unstale":"2024-04-25T14:02:14.182085+0000","last_undegraded":"2024-04-25T14:02:14.182085+0000","last_fullsized":"2024-04-25T14:02:14.182085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:02:32.599 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T14:02:32.600 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T14:02:32.600 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-25T14:02:32.600 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-25T14:02:32.600 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-25T14:02:32.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.742+0000 7fc9a3c0c640 1 Processor -- start 2024-04-25T14:02:32.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.742+0000 7fc9a3c0c640 1 -- start start 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a3c0c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c0d4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a3c0c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc99c06cf80 0x7fc99c0d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a3c0c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc99c0d14d0 con 0x7fc99c0d10c0 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a3c0c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc99c0b9680 con 0x7fc99c06cf80 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a3c0c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc99c06d790 con 0x7fc99c0b4e00 2024-04-25T14:02:32.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a19a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c0d4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a11a7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc99c06cf80 0x7fc99c0d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a19a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c0d4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:41908/0 (socket says 172.21.15.150:41908) 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a21a9640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fc99c0d10c0 0x7fc99c0d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:33112/0 (socket says 172.21.15.150:33112) 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a19a8640 1 -- 172.21.15.150:0/3701239812 learned_addr learned my addr 172.21.15.150:0/3701239812 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a11a7640 1 -- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 msgr2=0x7fc99c0d4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a11a7640 1 --2- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c0d4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a11a7640 1 -- 172.21.15.150:0/3701239812 >> v1:172.21.15.187:6789/0 conn(0x7fc99c0d10c0 legacy=0x7fc99c0d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.744+0000 7fc9a11a7640 1 -- 172.21.15.150:0/3701239812 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c06da70 con 0x7fc99c06cf80 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.745+0000 7fc9a11a7640 1 --2- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c0d0b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc98c0097a0 tx=0x7fc98c02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fc79be8d47dbbca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.745+0000 7fc9a09a6640 1 -- 172.21.15.150:0/3701239812 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c037070 con 0x7fc99c06cf80 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.745+0000 7fc9a09a6640 1 -- 172.21.15.150:0/3701239812 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c032440 con 0x7fc99c06cf80 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.745+0000 7fc9a09a6640 1 -- 172.21.15.150:0/3701239812 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c032810 con 0x7fc99c06cf80 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 msgr2=0x7fc99c0d0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c0d0b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc98c0097a0 tx=0x7fc98c02a210 comp rx=0 tx=0).stop 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/3701239812 shutdown_connections 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c0d0b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/3701239812 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c0d4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/3701239812 >> 172.21.15.150:0/3701239812 conn(0x7fc99c0d63d0 msgr2=0x7fc99c0d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/3701239812 shutdown_connections 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.746+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/3701239812 wait complete. 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.748+0000 7fc9a3c0c640 1 Processor -- start 2024-04-25T14:02:32.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.748+0000 7fc9a3c0c640 1 -- start start 2024-04-25T14:02:32.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c1fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c06d390 con 0x7fc99c0d10c0 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c0b92a0 con 0x7fc99c06cf80 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a3c0c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc99c06d510 con 0x7fc99c0b4e00 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a21a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a21a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38248/0 (socket says 172.21.15.150:38248) 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a21a9640 1 -- 172.21.15.150:0/2434628580 learned_addr learned my addr 172.21.15.150:0/2434628580 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a19a8640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.749+0000 7fc9a11a7640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c1fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 -- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 msgr2=0x7fc99c1fa280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c1fa280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 -- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 msgr2=0x7fc99c153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 -- 172.21.15.150:0/2434628580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c0cf720 con 0x7fc99c0d10c0 2024-04-25T14:02:32.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a21a9640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c680 tx=0x7fc99400cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc994018070 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc994004d60 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99c0cfa00 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99400a880 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.750+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99c0cec10 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.751+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc99c0cefd0 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.752+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc99400aa20 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.752+0000 7fc9927fc640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 0x7fc9800769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.752+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc9940949a0 con 0x7fc99c0d10c0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.752+0000 7fc9a19a8640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 0x7fc9800769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:32.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.752+0000 7fc9a19a8640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 0x7fc9800769c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc998009b30 tx=0x7fc998009240 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:32.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.757+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc99c0d14d0 con 0x7fc99c0d10c0 2024-04-25T14:02:32.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.984+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc99c076a40 con 0x7fc99c0d10c0 2024-04-25T14:02:32.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.987+0000 7fc9927fc640 1 -- 172.21.15.150:0/2434628580 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+186749 (secure 0 0 0) 0x7fc99405e7b0 con 0x7fc99c0d10c0 2024-04-25T14:02:32.988 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 msgr2=0x7fc9800769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 0x7fc9800769c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc998009b30 tx=0x7fc998009240 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 msgr2=0x7fc99c0cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc99400c680 tx=0x7fc99400cb40 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 shutdown_connections 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9800744b0 0x7fc9800769c0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d10c0 0x7fc99c0cb930 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c0b4e00 0x7fc99c1fa280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 --2- 172.21.15.150:0/2434628580 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c06cf80 0x7fc99c153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 >> 172.21.15.150:0/2434628580 conn(0x7fc99c0d63d0 msgr2=0x7fc99c0b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:32.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 shutdown_connections 2024-04-25T14:02:32.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:32.990+0000 7fc9a3c0c640 1 -- 172.21.15.150:0/2434628580 wait complete. 2024-04-25T14:02:33.000 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6832","nonce":3849822706},{"type":"v1","addr":"172.21.15.150:6833","nonce":3849822706}]},"active_addr":"172.21.15.150:6833/3849822706","active_change":"2024-04-25T14:01:48.577882+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":4294930464}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":317265765}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":1682610250}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.150:0","nonce":4215331925}]}]} 2024-04-25T14:02:33.003 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-25T14:02:33.004 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-25T14:02:33.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:33.110 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.111+0000 7efbfcc27640 1 Processor -- start 2024-04-25T14:02:33.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.111+0000 7efbfcc27640 1 -- start start 2024-04-25T14:02:33.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbfcc27640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efbf8161b10 0x7efbf8161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbfcc27640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efbf8154950 0x7efbf8154d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbfcc27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efbf8071600 con 0x7efbf80d1120 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbfcc27640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7efbf80d1530 con 0x7efbf8161b10 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbfcc27640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efbf8162460 con 0x7efbf8154950 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efbf80d1120 0x7efbf8070ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55490/0 (socket says 172.21.15.150:55490) 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf659c640 1 -- 172.21.15.150:0/3171991431 learned_addr learned my addr 172.21.15.150:0/3171991431 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf5d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efbf8161b10 0x7efbf8161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf6d9d640 1 --2- 172.21.15.150:0/3171991431 >> v2:172.21.15.187:3300/0 conn(0x7efbf8154950 0x7efbf8154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf5d9b640 1 -- 172.21.15.150:0/3171991431 >> v2:172.21.15.187:3300/0 conn(0x7efbf8154950 msgr2=0x7efbf8154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf5d9b640 1 --2- 172.21.15.150:0/3171991431 >> v2:172.21.15.187:3300/0 conn(0x7efbf8154950 0x7efbf8154d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf5d9b640 1 -- 172.21.15.150:0/3171991431 >> v1:172.21.15.150:6789/0 conn(0x7efbf80d1120 legacy=0x7efbf8070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf5d9b640 1 -- 172.21.15.150:0/3171991431 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf80aec30 con 0x7efbf8161b10 2024-04-25T14:02:33.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.113+0000 7efbf559a640 1 -- 172.21.15.150:0/3171991431 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4103504997 0 0) 0x7efbf8071600 con 0x7efbf80d1120 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.114+0000 7efbf5d9b640 1 --2- 172.21.15.150:0/3171991431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8161b10 0x7efbf8161f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efbf80730e0 tx=0x7efbec02a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2595a6f271eab66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.114+0000 7efbf559a640 1 -- 172.21.15.150:0/3171991431 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec037070 con 0x7efbf8161b10 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.114+0000 7efbf559a640 1 -- 172.21.15.150:0/3171991431 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbec02add0 con 0x7efbf8161b10 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.114+0000 7efbf559a640 1 -- 172.21.15.150:0/3171991431 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec032660 con 0x7efbf8161b10 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 -- 172.21.15.150:0/3171991431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8161b10 msgr2=0x7efbf8161f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 --2- 172.21.15.150:0/3171991431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8161b10 0x7efbf8161f20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7efbf80730e0 tx=0x7efbec02a080 comp rx=0 tx=0).stop 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 -- 172.21.15.150:0/3171991431 shutdown_connections 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 --2- 172.21.15.150:0/3171991431 >> v2:172.21.15.187:3300/0 conn(0x7efbf8154950 0x7efbf8154d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 --2- 172.21.15.150:0/3171991431 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8161b10 0x7efbf8161f20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 -- 172.21.15.150:0/3171991431 >> 172.21.15.150:0/3171991431 conn(0x7efbf80d1ae0 msgr2=0x7efbf80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 -- 172.21.15.150:0/3171991431 shutdown_connections 2024-04-25T14:02:33.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.115+0000 7efbfcc27640 1 -- 172.21.15.150:0/3171991431 wait complete. 2024-04-25T14:02:33.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.117+0000 7efbfcc27640 1 Processor -- start 2024-04-25T14:02:33.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.117+0000 7efbfcc27640 1 -- start start 2024-04-25T14:02:33.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf80d1120 0x7efbf81fad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efbf8161b10 0x7efbf81f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efbf80af4d0 con 0x7efbf8154950 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efbf80af1f0 con 0x7efbf80d1120 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbfcc27640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efbf81552a0 con 0x7efbf8161b10 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbf5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38274/0 (socket says 172.21.15.150:38274) 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.118+0000 7efbf5d9b640 1 -- 172.21.15.150:0/1553508814 learned_addr learned my addr 172.21.15.150:0/1553508814 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf659c640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf80d1120 0x7efbf81fad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf6d9d640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efbf8161b10 0x7efbf81f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 -- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efbf8161b10 msgr2=0x7efbf81f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efbf8161b10 0x7efbf81f6bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 -- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf80d1120 msgr2=0x7efbf81fad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf80d1120 0x7efbf81fad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 -- 172.21.15.150:0/1553508814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbf81576e0 con 0x7efbf8154950 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbf5d9b640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efbec009890 tx=0x7efbec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec037040 con 0x7efbf8154950 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbec042070 con 0x7efbf8154950 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efbf81f7790 con 0x7efbf8154950 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbec03d660 con 0x7efbf8154950 2024-04-25T14:02:33.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.119+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efbf81fa540 con 0x7efbf8154950 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.120+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efbf8076a40 con 0x7efbf8154950 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.121+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efbec03d800 con 0x7efbf8154950 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.121+0000 7efbe6ffd640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 0x7efbd80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.121+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7efbec0b5920 con 0x7efbf8154950 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.121+0000 7efbf659c640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 0x7efbd80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.122+0000 7efbf659c640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 0x7efbd80769f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efbe80012e0 tx=0x7efbe8001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.126+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efbec07f7b0 con 0x7efbf8154950 2024-04-25T14:02:33.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.307+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efbf8070250 con 0x7efbf8154950 2024-04-25T14:02:33.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.308+0000 7efbe6ffd640 1 -- 172.21.15.150:0/1553508814 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7efbec083ab0 con 0x7efbf8154950 2024-04-25T14:02:33.308 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:33.308 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 msgr2=0x7efbd80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 0x7efbd80769f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efbe80012e0 tx=0x7efbe8001630 comp rx=0 tx=0).stop 2024-04-25T14:02:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 msgr2=0x7efbf81f0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efbec009890 tx=0x7efbec0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 shutdown_connections 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efbd80744e0 0x7efbd80769f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efbf8161b10 0x7efbf81f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efbf8154950 0x7efbf81f0be0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 --2- 172.21.15.150:0/1553508814 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efbf80d1120 0x7efbf81fad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 >> 172.21.15.150:0/1553508814 conn(0x7efbf80d1ae0 msgr2=0x7efbf80ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 shutdown_connections 2024-04-25T14:02:33.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.311+0000 7efbfcc27640 1 -- 172.21.15.150:0/1553508814 wait complete. 2024-04-25T14:02:33.321 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-25T14:02:33.322 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:33.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.429+0000 7f922ba92640 1 Processor -- start 2024-04-25T14:02:33.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.430+0000 7f922ba92640 1 -- start start 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922ba92640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240d4580 0x7f92240b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922ba92640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92240d1060 0x7f92240b9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922ba92640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f922406cb10 con 0x7f922406c570 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922ba92640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f922406cdf0 con 0x7f92240d4580 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922ba92640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f922406d0d0 con 0x7f92240d1060 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240d4580 0x7f92240b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92240d4580 0x7f92240b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38280/0 (socket says 172.21.15.150:38280) 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 -- 172.21.15.150:0/2616963361 learned_addr learned my addr 172.21.15.150:0/2616963361 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922a02f640 1 --2- 172.21.15.150:0/2616963361 >> v2:172.21.15.187:3300/0 conn(0x7f92240d1060 0x7f92240b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 -- 172.21.15.150:0/2616963361 >> v2:172.21.15.187:3300/0 conn(0x7f92240d1060 msgr2=0x7f92240b9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 --2- 172.21.15.150:0/2616963361 >> v2:172.21.15.187:3300/0 conn(0x7f92240d1060 0x7f92240b9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 -- 172.21.15.150:0/2616963361 >> v1:172.21.15.187:6789/0 conn(0x7f922406c570 legacy=0x7f92240d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.431+0000 7f922982e640 1 -- 172.21.15.150:0/2616963361 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922406d3b0 con 0x7f92240d4580 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.432+0000 7f922982e640 1 --2- 172.21.15.150:0/2616963361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240d4580 0x7f92240b4df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f922408efc0 tx=0x7f922002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8ff90c04107593 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.432+0000 7f922882c640 1 -- 172.21.15.150:0/2616963361 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220036070 con 0x7f92240d4580 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.432+0000 7f922882c640 1 -- 172.21.15.150:0/2616963361 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9220031600 con 0x7f92240d4580 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.432+0000 7f922882c640 1 -- 172.21.15.150:0/2616963361 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92200319d0 con 0x7f92240d4580 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 -- 172.21.15.150:0/2616963361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240d4580 msgr2=0x7f92240b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 --2- 172.21.15.150:0/2616963361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240d4580 0x7f92240b4df0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f922408efc0 tx=0x7f922002a730 comp rx=0 tx=0).stop 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 -- 172.21.15.150:0/2616963361 shutdown_connections 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 --2- 172.21.15.150:0/2616963361 >> v2:172.21.15.187:3300/0 conn(0x7f92240d1060 0x7f92240b9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 --2- 172.21.15.150:0/2616963361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92240d4580 0x7f92240b4df0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 -- 172.21.15.150:0/2616963361 >> 172.21.15.150:0/2616963361 conn(0x7f92240d6470 msgr2=0x7f92240d2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 -- 172.21.15.150:0/2616963361 shutdown_connections 2024-04-25T14:02:33.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.433+0000 7f922ba92640 1 -- 172.21.15.150:0/2616963361 wait complete. 2024-04-25T14:02:33.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.435+0000 7f922ba92640 1 Processor -- start 2024-04-25T14:02:33.434 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.435+0000 7f922ba92640 1 -- start start 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 0x7f92241f60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 0x7f9224159c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f92240d1670 con 0x7f922406c570 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f92240d14f0 con 0x7f92241f6620 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922ba92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f922406dbb0 con 0x7f92240d4580 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922982e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922982e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38290/0 (socket says 172.21.15.150:38290) 2024-04-25T14:02:33.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922982e640 1 -- 172.21.15.150:0/1115541838 learned_addr learned my addr 172.21.15.150:0/1115541838 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922a02f640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 0x7f9224159c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.436+0000 7f922902d640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 0x7f92241f60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 -- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 msgr2=0x7f92241f60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 0x7f92241f60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 -- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 msgr2=0x7f9224159c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 0x7f9224159c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 -- 172.21.15.150:0/1115541838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f922414e370 con 0x7f922406c570 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922a02f640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 0x7f9224159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922902d640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 0x7f92241f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922982e640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9220009f40 tx=0x7f9220004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f922003f020 con 0x7f922406c570 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9220037040 con 0x7f922406c570 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92241580d0 con 0x7f922406c570 2024-04-25T14:02:33.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220036070 con 0x7f922406c570 2024-04-25T14:02:33.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.437+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92240cba70 con 0x7f922406c570 2024-04-25T14:02:33.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.438+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9224076a40 con 0x7f922406c570 2024-04-25T14:02:33.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.439+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9220042020 con 0x7f922406c570 2024-04-25T14:02:33.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.439+0000 7f921a7fc640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 0x7f92080769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.439+0000 7f922902d640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 0x7f92080769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.439+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f92200b5840 con 0x7f922406c570 2024-04-25T14:02:33.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.439+0000 7f922902d640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 0x7f92080769c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f922415f100 tx=0x7f9214009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.444+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9220049070 con 0x7f922406c570 2024-04-25T14:02:33.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.621+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92240d3f80 con 0x7f922406c570 2024-04-25T14:02:33.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.622+0000 7f921a7fc640 1 -- 172.21.15.150:0/1115541838 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f922007fd50 con 0x7f922406c570 2024-04-25T14:02:33.622 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:33.622 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 msgr2=0x7f92080769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 0x7f92080769c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f922415f100 tx=0x7f9214009040 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 msgr2=0x7f922414de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9220009f40 tx=0x7f9220004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 shutdown_connections 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92080744b0 0x7f92080769c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92241f6620 0x7f9224159c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92240d4580 0x7f92241f60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 --2- 172.21.15.150:0/1115541838 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f922406c570 0x7f922414de30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 >> 172.21.15.150:0/1115541838 conn(0x7f92240d6470 msgr2=0x7f92241622d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 shutdown_connections 2024-04-25T14:02:33.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.624+0000 7f922ba92640 1 -- 172.21.15.150:0/1115541838 wait complete. 2024-04-25T14:02:33.635 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-25T14:02:33.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-25T14:02:33.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-25T14:02:33.636 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-25T14:02:33.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-25T14:02:33.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-25T14:02:33.637 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-25T14:02:33.638 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-25T14:02:33.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7f164759e640 1 Processor -- start 2024-04-25T14:02:33.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7f4062fa8640 1 Processor -- start 2024-04-25T14:02:33.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7f164759e640 1 -- start start 2024-04-25T14:02:33.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7fdc2832d640 1 Processor -- start 2024-04-25T14:02:33.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7fdc2832d640 1 -- start start 2024-04-25T14:02:33.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.863+0000 7f4062fa8640 1 -- start start 2024-04-25T14:02:33.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7ff06d23d640 1 Processor -- start 2024-04-25T14:02:33.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7f3c6e596640 1 Processor -- start 2024-04-25T14:02:33.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7ff8f8e97640 1 Processor -- start 2024-04-25T14:02:33.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7ff8f8e97640 1 -- start start 2024-04-25T14:02:33.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7f3c6e596640 1 -- start start 2024-04-25T14:02:33.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.865+0000 7f3fa139d640 1 Processor -- start 2024-04-25T14:02:33.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.865+0000 7f3fa139d640 1 -- start start 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc2832d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc1803a5b0 0x7fdc180e5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc260c9640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc1803a5b0 0x7fdc180e5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc260c9640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc1803a5b0 0x7fdc180e5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43144/0 (socket says 172.21.15.150:43144) 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc258c8640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdc180384f0 0x7fdc18038900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:33148/0 (socket says 172.21.15.150:33148) 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc260c9640 1 -- 172.21.15.150:0/308823818 learned_addr learned my addr 172.21.15.150:0/308823818 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc1803a9c0 con 0x7fdc180060c0 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc1803ac70 con 0x7fdc180384f0 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc180e1c20 con 0x7fdc1803a5b0 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc250c7640 1 -- 172.21.15.150:0/308823818 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1096667215 0 0) 0x7fdc1803ac70 con 0x7fdc180384f0 2024-04-25T14:02:33.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc250c7640 1 -- 172.21.15.150:0/308823818 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc0c003520 con 0x7fdc180384f0 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f164759e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f16400340c0 0x7f1640006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f164759e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16400071d0 con 0x7f16400e08b0 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f164759e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16400074b0 con 0x7f16400dfd90 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f164759e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f16400344d0 con 0x7f16400340c0 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f164659c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f16400340c0 0x7f1640006aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f1646d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f16400e08b0 0x7f164004fa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55506/0 (socket says 172.21.15.150:55506) 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f1646d9d640 1 -- 172.21.15.150:0/1346421334 learned_addr learned my addr 172.21.15.150:0/1346421334 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c6e596640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c6e596640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6815e3d0 con 0x7f3c6809b4a0 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c6e596640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c680b3af0 con 0x7f3c680b36e0 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c6e596640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3c680b3250 con 0x7f3c681576d0 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c67fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c67fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:41942/0 (socket says 172.21.15.150:41942) 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f3c67fff640 1 -- 172.21.15.150:0/1992123388 learned_addr learned my addr 172.21.15.150:0/1992123388 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3c67fff640 1 -- 172.21.15.150:0/1992123388 >> v1:172.21.15.187:6789/0 conn(0x7f3c680b36e0 legacy=0x7f3c6815db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.865+0000 7f5d01fce640 1 Processor -- start 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.865+0000 7f5d01fce640 1 -- start start 2024-04-25T14:02:33.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf4006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.865+0000 7f4062fa8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4054047db0 0x7f40540e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4062fa8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4054006130 con 0x7f4054034060 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4062fa8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f405404ffa0 con 0x7f4054048300 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4062fa8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f405404f690 con 0x7f4054047db0 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4060d44640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4054047db0 0x7f40540e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4060d44640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4054047db0 0x7f40540e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38302/0 (socket says 172.21.15.150:38302) 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.866+0000 7f4060d44640 1 -- 172.21.15.150:0/4114374438 learned_addr learned my addr 172.21.15.150:0/4114374438 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f4060d44640 1 -- 172.21.15.150:0/4114374438 >> v1:172.21.15.187:6789/0 conn(0x7f4054048300 legacy=0x7f4054033950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f4060d44640 1 -- 172.21.15.150:0/4114374438 >> v1:172.21.15.150:6789/0 conn(0x7f4054034060 legacy=0x7f40540065e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7f4060d44640 1 -- 172.21.15.150:0/4114374438 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4054050280 con 0x7f4054047db0 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.864+0000 7ff06d23d640 1 -- start start 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff060047db0 0x7ff0600e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060048300 0x7ff060033950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff060033e90 0x7ff0600531e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff06004f690 con 0x7ff060033e90 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff060006130 con 0x7ff060047db0 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06d23d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0600342a0 con 0x7ff060048300 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff067fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff060047db0 0x7ff0600e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff067fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff060047db0 0x7ff0600e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43152/0 (socket says 172.21.15.150:43152) 2024-04-25T14:02:33.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff067fff640 1 -- 172.21.15.150:0/3780947348 learned_addr learned my addr 172.21.15.150:0/3780947348 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff0677fe640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060048300 0x7ff060033950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff06ca3c640 1 --2- 172.21.15.150:0/3780947348 >> v2:172.21.15.150:3300/0 conn(0x7ff060033e90 0x7ff0600531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff8f8e97640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8ec0340c0 0x7ff8ec006aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff8f8e97640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0e08b0 0x7ff8ec04fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff8f8e97640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8ec0071d0 con 0x7ff8ec0dfd90 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff8f8e97640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff8ec0344d0 con 0x7ff8ec0340c0 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7ff8f8e97640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ec0074b0 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff8f3fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0e08b0 0x7ff8ec04fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3c67fff640 1 -- 172.21.15.150:0/1992123388 >> v1:172.21.15.150:6789/0 conn(0x7f3c6809b4a0 legacy=0x7f3c6809b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8ec0340c0 0x7ff8ec006aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff8f37fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff8ec0340c0 0x7ff8ec006aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43154/0 (socket says 172.21.15.150:43154) 2024-04-25T14:02:33.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff8f37fe640 1 -- 172.21.15.150:0/2065627638 learned_addr learned my addr 172.21.15.150:0/2065627638 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff067fff640 1 -- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060048300 msgr2=0x7ff060033950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5cf40dfd90 0x7f5cf40e01a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3fa139d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f9c0cfff0 0x7f3f9c15dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3fa139d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 0x7f3f9c06d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3fa139d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f9c15e2e0 con 0x7f3f9c06dac0 2024-04-25T14:02:33.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3fa139d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3f9c0d1200 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3fa139d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3f9c06ded0 con 0x7f3f9c143040 2024-04-25T14:02:33.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc260c9640 1 -- 172.21.15.150:0/308823818 >> v1:172.21.15.187:6789/0 conn(0x7fdc180384f0 legacy=0x7fdc18038900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc260c9640 1 -- 172.21.15.150:0/308823818 >> v1:172.21.15.150:6789/0 conn(0x7fdc180060c0 legacy=0x7fdc180064d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.867+0000 7fdc260c9640 1 -- 172.21.15.150:0/308823818 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc180e1f00 con 0x7fdc1803a5b0 2024-04-25T14:02:33.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 0x7f5cf40e3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff067fff640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060048300 0x7ff060033950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff067fff640 1 -- 172.21.15.150:0/3780947348 >> v2:172.21.15.150:3300/0 conn(0x7ff060033e90 msgr2=0x7ff0600531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7fdc260c9640 1 --2- 172.21.15.150:0/308823818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1803a5b0 0x7fdc180e5b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdc2006bd20 tx=0x7fdc200a7190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69877b2d5533567f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3f9b7fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 0x7f3f9c06d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3f9b7fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 0x7f3f9c06d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43162/0 (socket says 172.21.15.150:43162) 2024-04-25T14:02:33.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3f9b7fe640 1 -- 172.21.15.150:0/1684757207 learned_addr learned my addr 172.21.15.150:0/1684757207 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d017cd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 0x7f5cf40e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff067fff640 1 --2- 172.21.15.150:0/3780947348 >> v2:172.21.15.150:3300/0 conn(0x7ff060033e90 0x7ff0600531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff067fff640 1 -- 172.21.15.150:0/3780947348 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff06004ffa0 con 0x7ff060047db0 2024-04-25T14:02:33.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d017cd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 0x7f5cf40e3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38326/0 (socket says 172.21.15.150:38326) 2024-04-25T14:02:33.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3c67fff640 1 -- 172.21.15.150:0/1992123388 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c680b2920 con 0x7f3c681576d0 2024-04-25T14:02:33.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d017cd640 1 -- 172.21.15.150:0/2067343498 learned_addr learned my addr 172.21.15.150:0/2067343498 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf40344d0 con 0x7f5cf40e06e0 2024-04-25T14:02:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5cf40071d0 con 0x7f5cf40dfd90 2024-04-25T14:02:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7fdc250c7640 1 -- 172.21.15.150:0/308823818 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200b4070 con 0x7fdc1803a5b0 2024-04-25T14:02:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf40074b0 con 0x7f5cf40340c0 2024-04-25T14:02:33.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7ff8f37fe640 1 -- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0e08b0 msgr2=0x7ff8ec04fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7fdc250c7640 1 -- 172.21.15.150:0/308823818 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc200a7cf0 con 0x7fdc1803a5b0 2024-04-25T14:02:33.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7fdc250c7640 1 -- 172.21.15.150:0/308823818 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200af640 con 0x7fdc1803a5b0 2024-04-25T14:02:33.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/2067343498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf4006aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5cfbfff640 1 --2- 172.21.15.150:0/2067343498 >> v2:172.21.15.187:3300/0 conn(0x7f5cf40dfd90 0x7f5cf40e01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f164559a640 1 -- 172.21.15.150:0/1346421334 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3373158931 0 0) 0x7f16400074b0 con 0x7f16400dfd90 2024-04-25T14:02:33.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f164659c640 1 -- 172.21.15.150:0/1346421334 >> v1:172.21.15.187:6789/0 conn(0x7f16400dfd90 legacy=0x7f16400e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f164659c640 1 -- 172.21.15.150:0/1346421334 >> v1:172.21.15.150:6789/0 conn(0x7f16400e08b0 legacy=0x7f164004fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f164659c640 1 -- 172.21.15.150:0/1346421334 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1640007790 con 0x7f16400340c0 2024-04-25T14:02:33.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff8f37fe640 1 --2- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0e08b0 0x7ff8ec04fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff8f37fe640 1 -- 172.21.15.150:0/2065627638 >> v1:172.21.15.150:6789/0 conn(0x7ff8ec0dfd90 legacy=0x7ff8ec0e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff8f37fe640 1 -- 172.21.15.150:0/2065627638 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec007790 con 0x7ff8ec0340c0 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5d00fcc640 1 -- 172.21.15.150:0/2067343498 >> v2:172.21.15.187:3300/0 conn(0x7f5cf40dfd90 msgr2=0x7f5cf40e01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/2067343498 >> v2:172.21.15.187:3300/0 conn(0x7f5cf40dfd90 0x7f5cf40e01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5d00fcc640 1 -- 172.21.15.150:0/2067343498 >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 msgr2=0x7f5cf40e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/2067343498 >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 0x7f5cf40e3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f5d00fcc640 1 -- 172.21.15.150:0/2067343498 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf4007790 con 0x7f5cf40340c0 2024-04-25T14:02:33.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff067fff640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff0600e7d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d9b0 tx=0x7ff05c02a1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c9d556e8832feea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.868+0000 7f3f9bfff640 1 --2- 172.21.15.150:0/1684757207 >> v2:172.21.15.150:3300/0 conn(0x7f3f9c0cfff0 0x7f3f9c15dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3c67fff640 1 --2- 172.21.15.150:0/1992123388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009e80 tx=0x7f3c6002a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72c2cd7b5b775629 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9affd640 1 -- 172.21.15.150:0/1684757207 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776193035 0 0) 0x7f3f94001490 con 0x7f3f9c06dac0 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9affd640 1 -- 172.21.15.150:0/1684757207 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f88003520 con 0x7f3f9c06dac0 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff066ffd640 1 -- 172.21.15.150:0/3780947348 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c037070 con 0x7ff060047db0 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff066ffd640 1 -- 172.21.15.150:0/3780947348 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05c032440 con 0x7ff060047db0 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7ff066ffd640 1 -- 172.21.15.150:0/3780947348 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0327f0 con 0x7ff060047db0 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9bfff640 1 -- 172.21.15.150:0/1684757207 >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 msgr2=0x7f3f9c06d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9bfff640 1 --2- 172.21.15.150:0/1684757207 >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 0x7f3f9c06d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9bfff640 1 -- 172.21.15.150:0/1684757207 >> v1:172.21.15.187:6789/0 conn(0x7f3f9c06dac0 legacy=0x7f3f9c160910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f3f9bfff640 1 -- 172.21.15.150:0/1684757207 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c0d05f0 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f4060d44640 1 --2- 172.21.15.150:0/4114374438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f40540e7d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f405404f810 tx=0x7f405002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f8446bf2c75e47 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.869+0000 7f164659c640 1 --2- 172.21.15.150:0/1346421334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400340c0 0x7f1640006aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f16380071d0 tx=0x7f163802c2b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4be223bda7a9a1b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1803a5b0 msgr2=0x7fdc180e5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 --2- 172.21.15.150:0/308823818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1803a5b0 0x7fdc180e5b10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdc2006bd20 tx=0x7fdc200a7190 comp rx=0 tx=0).stop 2024-04-25T14:02:33.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f164559a640 1 -- 172.21.15.150:0/1346421334 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380089e0 con 0x7f16400340c0 2024-04-25T14:02:33.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff8f37fe640 1 --2- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0340c0 0x7ff8ec006aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009870 tx=0x7ff8e802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d015e37159c01f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/2067343498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf4006aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40378e0 tx=0x7f5cf002e710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55f166585f283dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f164559a640 1 -- 172.21.15.150:0/1346421334 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1638008b80 con 0x7f16400340c0 2024-04-25T14:02:33.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f3c66ffd640 1 -- 172.21.15.150:0/1992123388 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60036070 con 0x7f3c681576d0 2024-04-25T14:02:33.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 shutdown_connections 2024-04-25T14:02:33.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f3c66ffd640 1 -- 172.21.15.150:0/1992123388 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c60031600 con 0x7f3c681576d0 2024-04-25T14:02:33.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f3c66ffd640 1 -- 172.21.15.150:0/1992123388 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c600319d0 con 0x7f3c681576d0 2024-04-25T14:02:33.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 --2- 172.21.15.150:0/308823818 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc1803a5b0 0x7fdc180e5b10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f164559a640 1 -- 172.21.15.150:0/1346421334 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163800a0d0 con 0x7f16400340c0 2024-04-25T14:02:33.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 >> 172.21.15.150:0/308823818 conn(0x7fdc18054d20 msgr2=0x7fdc18055130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 shutdown_connections 2024-04-25T14:02:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff8f27fc640 1 -- 172.21.15.150:0/2065627638 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8037070 con 0x7ff8ec0340c0 2024-04-25T14:02:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff8f27fc640 1 -- 172.21.15.150:0/2065627638 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8e8032440 con 0x7ff8ec0340c0 2024-04-25T14:02:33.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7fdc2832d640 1 -- 172.21.15.150:0/308823818 wait complete. 2024-04-25T14:02:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff8f27fc640 1 -- 172.21.15.150:0/2065627638 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8032810 con 0x7ff8ec0340c0 2024-04-25T14:02:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f405b7fe640 1 -- 172.21.15.150:0/4114374438 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050037070 con 0x7f4054047db0 2024-04-25T14:02:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f405b7fe640 1 -- 172.21.15.150:0/4114374438 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4050032440 con 0x7f4054047db0 2024-04-25T14:02:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f405b7fe640 1 -- 172.21.15.150:0/4114374438 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4050037070 con 0x7f4054047db0 2024-04-25T14:02:33.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff06d23d640 1 -- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 msgr2=0x7ff0600e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7ff06d23d640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff0600e7d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d9b0 tx=0x7ff05c02a1f0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.870+0000 7f3f9bfff640 1 --2- 172.21.15.150:0/1684757207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c0cfff0 0x7f3f9c15dda0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c157970 tx=0x7f3f8402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de998f1beca226ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 -- 172.21.15.150:0/3780947348 shutdown_connections 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 --2- 172.21.15.150:0/3780947348 >> v2:172.21.15.150:3300/0 conn(0x7ff060033e90 0x7ff0600531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060048300 0x7ff060033950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 --2- 172.21.15.150:0/3780947348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff0600e7d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 -- 172.21.15.150:0/3780947348 >> 172.21.15.150:0/3780947348 conn(0x7ff060054cf0 msgr2=0x7ff060055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 -- 172.21.15.150:0/3780947348 shutdown_connections 2024-04-25T14:02:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f3f9affd640 1 -- 172.21.15.150:0/1684757207 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f84037070 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f3f9affd640 1 -- 172.21.15.150:0/1684757207 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f84032440 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f3f9affd640 1 -- 172.21.15.150:0/1684757207 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8402ae60 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff06d23d640 1 -- 172.21.15.150:0/3780947348 wait complete. 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 -- 172.21.15.150:0/1346421334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400340c0 msgr2=0x7f1640006aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 --2- 172.21.15.150:0/1346421334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400340c0 0x7f1640006aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f16380071d0 tx=0x7f163802c2b0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f3c6e596640 1 -- 172.21.15.150:0/1992123388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 msgr2=0x7f3c680d3320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f3c6e596640 1 --2- 172.21.15.150:0/1992123388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009e80 tx=0x7f3c6002a6c0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 -- 172.21.15.150:0/1346421334 shutdown_connections 2024-04-25T14:02:33.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 --2- 172.21.15.150:0/1346421334 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400340c0 0x7f1640006aa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 -- 172.21.15.150:0/1346421334 >> 172.21.15.150:0/1346421334 conn(0x7f1640054cf0 msgr2=0x7f1640055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff8f8e97640 1 -- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0340c0 msgr2=0x7ff8ec006aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0340c0 0x7ff8ec006aa0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8009870 tx=0x7ff8e802a180 comp rx=0 tx=0).stop 2024-04-25T14:02:33.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f164759e640 1 -- 172.21.15.150:0/1346421334 shutdown_connections 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7f5cfb7fe640 1 -- 172.21.15.150:0/2067343498 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf003c070 con 0x7f5cf40340c0 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff8f8e97640 1 -- 172.21.15.150:0/2065627638 shutdown_connections 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0e08b0 0x7ff8ec04fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.871+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/2065627638 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0340c0 0x7ff8ec006aa0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7ff8f8e97640 1 -- 172.21.15.150:0/2065627638 >> 172.21.15.150:0/2065627638 conn(0x7ff8ec054cf0 msgr2=0x7ff8ec055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f5cfb7fe640 1 -- 172.21.15.150:0/2067343498 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0034480 con 0x7f5cf40340c0 2024-04-25T14:02:33.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7ff8f8e97640 1 -- 172.21.15.150:0/2065627638 shutdown_connections 2024-04-25T14:02:33.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f5cfb7fe640 1 -- 172.21.15.150:0/2067343498 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0034870 con 0x7f5cf40340c0 2024-04-25T14:02:33.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7ff8f8e97640 1 -- 172.21.15.150:0/2065627638 wait complete. 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 -- 172.21.15.150:0/1684757207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c0cfff0 msgr2=0x7f3f9c15dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 --2- 172.21.15.150:0/1684757207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c0cfff0 0x7f3f9c15dda0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c157970 tx=0x7f3f8402a110 comp rx=0 tx=0).stop 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 -- 172.21.15.150:0/1684757207 shutdown_connections 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 --2- 172.21.15.150:0/1684757207 >> v2:172.21.15.187:3300/0 conn(0x7f3f9c143040 0x7f3f9c06d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 --2- 172.21.15.150:0/1684757207 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c0cfff0 0x7f3f9c15dda0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 -- 172.21.15.150:0/1684757207 >> 172.21.15.150:0/1684757207 conn(0x7f3f9c073090 msgr2=0x7f3f9c0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 -- 172.21.15.150:0/1684757207 shutdown_connections 2024-04-25T14:02:33.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f3fa139d640 1 -- 172.21.15.150:0/1684757207 wait complete. 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f4062fa8640 1 -- 172.21.15.150:0/4114374438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 msgr2=0x7f40540e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f4062fa8640 1 --2- 172.21.15.150:0/4114374438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f40540e7d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f405404f810 tx=0x7f405002a180 comp rx=0 tx=0).stop 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.872+0000 7f164759e640 1 -- 172.21.15.150:0/1346421334 wait complete. 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f3c6e596640 1 -- 172.21.15.150:0/1992123388 shutdown_connections 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f3c6e596640 1 --2- 172.21.15.150:0/1992123388 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c681576d0 0x7f3c680d3320 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f3c6e596640 1 -- 172.21.15.150:0/1992123388 >> 172.21.15.150:0/1992123388 conn(0x7f3c680b76d0 msgr2=0x7f3c680b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 msgr2=0x7f5cf4006aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 --2- 172.21.15.150:0/2067343498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf4006aa0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5cf40378e0 tx=0x7f5cf002e710 comp rx=0 tx=0).stop 2024-04-25T14:02:33.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f4062fa8640 1 -- 172.21.15.150:0/4114374438 shutdown_connections 2024-04-25T14:02:33.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f4062fa8640 1 --2- 172.21.15.150:0/4114374438 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f40540e7d50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f4062fa8640 1 -- 172.21.15.150:0/4114374438 >> 172.21.15.150:0/4114374438 conn(0x7f4054054cf0 msgr2=0x7f4054055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 shutdown_connections 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 --2- 172.21.15.150:0/2067343498 >> v2:172.21.15.150:3300/0 conn(0x7f5cf40e06e0 0x7f5cf40e3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 --2- 172.21.15.150:0/2067343498 >> v2:172.21.15.187:3300/0 conn(0x7f5cf40dfd90 0x7f5cf40e01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 --2- 172.21.15.150:0/2067343498 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf4006aa0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 >> 172.21.15.150:0/2067343498 conn(0x7f5cf4054cf0 msgr2=0x7f5cf4055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f4062fa8640 1 -- 172.21.15.150:0/4114374438 shutdown_connections 2024-04-25T14:02:33.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7fdc2832d640 1 Processor -- start 2024-04-25T14:02:33.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f3c6e596640 1 -- 172.21.15.150:0/1992123388 shutdown_connections 2024-04-25T14:02:33.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7fdc2832d640 1 -- start start 2024-04-25T14:02:33.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7ff06d23d640 1 Processor -- start 2024-04-25T14:02:33.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7f3c6e596640 1 -- 172.21.15.150:0/1992123388 wait complete. 2024-04-25T14:02:33.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7ff06d23d640 1 -- start start 2024-04-25T14:02:33.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 shutdown_connections 2024-04-25T14:02:33.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7f5d01fce640 1 -- 172.21.15.150:0/2067343498 wait complete. 2024-04-25T14:02:33.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.874+0000 7ff8f8e97640 1 Processor -- start 2024-04-25T14:02:33.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7f164759e640 1 Processor -- start 2024-04-25T14:02:33.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7ff8f8e97640 1 -- start start 2024-04-25T14:02:33.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7f164759e640 1 -- start start 2024-04-25T14:02:33.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7f3fa139d640 1 Processor -- start 2024-04-25T14:02:33.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc180060c0 0x7fdc18053e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180384f0 0x7fdc180daac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc1803a5b0 0x7fdc180d20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7f3fa139d640 1 -- start start 2024-04-25T14:02:33.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdc180e29e0 con 0x7fdc180060c0 2024-04-25T14:02:33.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc180e24c0 con 0x7fdc180384f0 2024-04-25T14:02:33.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.875+0000 7fdc2832d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc180e17d0 con 0x7fdc1803a5b0 2024-04-25T14:02:33.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.873+0000 7f4062fa8640 1 -- 172.21.15.150:0/4114374438 wait complete. 2024-04-25T14:02:33.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7f4062fa8640 1 Processor -- start 2024-04-25T14:02:33.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7fdc258c8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180384f0 0x7fdc180daac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7fdc260c9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc180060c0 0x7fdc18053e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7f4062fa8640 1 -- start start 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7fdc258c8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180384f0 0x7fdc180daac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43180/0 (socket says 172.21.15.150:43180) 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7fdc258c8640 1 -- 172.21.15.150:0/2126394628 learned_addr learned my addr 172.21.15.150:0/2126394628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.876+0000 7fdc268ca640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc1803a5b0 0x7fdc180d20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060033e90 0x7ff0600eae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff06004deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff060048300 0x7ff06004e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff060050840 con 0x7ff060048300 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff060050560 con 0x7ff060047db0 2024-04-25T14:02:33.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06d23d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff0600509c0 con 0x7ff060033e90 2024-04-25T14:02:33.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7fdc260c9640 1 -- 172.21.15.150:0/2126394628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc1803a5b0 msgr2=0x7fdc180d20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff067fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060033e90 0x7ff0600eae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff067fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060033e90 0x7ff0600eae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:41998/0 (socket says 172.21.15.150:41998) 2024-04-25T14:02:33.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff067fff640 1 -- 172.21.15.150:0/2143857121 learned_addr learned my addr 172.21.15.150:0/2143857121 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff06ca3c640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff060048300 0x7ff06004e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff8f8e97640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0340c0 0x7ff8ec1689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7fdc260c9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc1803a5b0 0x7fdc180d20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7fdc260c9640 1 -- 172.21.15.150:0/2126394628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180384f0 msgr2=0x7fdc180daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7fdc260c9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc180384f0 0x7fdc180daac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7fdc260c9640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc18052990 con 0x7fdc180060c0 2024-04-25T14:02:33.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff8f8e97640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 0x7ff8ec16a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff0677fe640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff06004deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0340c0 0x7ff8ec1689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f37fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0340c0 0x7ff8ec1689e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42012/0 (socket says 172.21.15.150:42012) 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16400340c0 0x7f16400d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400dfd90 0x7f16400d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16400e08b0 0x7f1640169610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1640050110 con 0x7f16400e08b0 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f164000c520 con 0x7f16400dfd90 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164759e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1640050290 con 0x7f16400340c0 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3fa139d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c06dac0 0x7f3f9c15a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3f9bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c06dac0 0x7f3f9c15a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7f5d01fce640 1 Processor -- start 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7f5d01fce640 1 -- start start 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f405404e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4054048300 0x7f40540e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405404cf70 0x7f405404ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f40540509c0 con 0x7f4054047db0 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4054050840 con 0x7f4054048300 2024-04-25T14:02:33.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4062fa8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4054006cf0 con 0x7f405404cf70 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 -- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060033e90 msgr2=0x7ff0600eae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff060033e90 0x7ff0600eae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 -- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 msgr2=0x7ff06004deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff060047db0 0x7ff06004deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff060049490 con 0x7ff060048300 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06ca3c640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff060048300 0x7ff06004e3f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff068066310 tx=0x7ff068069ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff068075820 con 0x7ff060048300 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff060049770 con 0x7ff060048300 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff060049d30 con 0x7ff060048300 2024-04-25T14:02:33.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 0x7ff8ec16a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f2ffd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 0x7ff8ec16a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43196/0 (socket says 172.21.15.150:43196) 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f2ffd640 1 -- 172.21.15.150:0/1362142257 learned_addr learned my addr 172.21.15.150:0/1362142257 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8ec0e08b0 0x7ff8ec0479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec04ff40 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8ec00c4e0 con 0x7ff8ec0dfd90 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8ec0500c0 con 0x7ff8ec0340c0 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff8f3fff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8ec0e08b0 0x7ff8ec0479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 -- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0340c0 msgr2=0x7ff8ec1689e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8ec0340c0 0x7ff8ec1689e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 -- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 msgr2=0x7ff8ec16a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 0x7ff8ec16a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8ec168f20 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f2ffd640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8ec0dfd90 0x7ff8ec16a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3f9bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c06dac0 0x7f3f9c15a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38356/0 (socket says 172.21.15.150:38356) 2024-04-25T14:02:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3f9bfff640 1 -- 172.21.15.150:0/3612083295 learned_addr learned my addr 172.21.15.150:0/3612083295 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3fa139d640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f9c0cfff0 0x7f3f9c141170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164659c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16400340c0 0x7f16400d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3fa139d640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f9c143040 0x7f3f9c0c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164659c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16400340c0 0x7f16400d1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42024/0 (socket says 172.21.15.150:42024) 2024-04-25T14:02:33.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f164659c640 1 -- 172.21.15.150:0/2907058320 learned_addr learned my addr 172.21.15.150:0/2907058320 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f1645d9b640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400dfd90 0x7f16400d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7f3c6e596640 1 Processor -- start 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.877+0000 7f3c6e596640 1 -- start start 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 0x7f3c68156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c680b36e0 0x7f3c6815a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c681576d0 0x7f3c68202ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6806e120 con 0x7f3c681576d0 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3c680b2ee0 con 0x7f3c680b36e0 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c6e596640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6806e2a0 con 0x7f3c6809b4a0 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c67fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 0x7f3c68156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c67fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 0x7f3c68156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42032/0 (socket says 172.21.15.150:42032) 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c67fff640 1 -- 172.21.15.150:0/3610475199 learned_addr learned my addr 172.21.15.150:0/3610475199 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f3c677fe640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c680b36e0 0x7f3c6815a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c0d0e90 con 0x7f3f9c06dac0 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9c0d0bb0 con 0x7f3f9c143040 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3f9c143450 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f9c0cfff0 0x7f3f9c141170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7f4061545640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405404cf70 0x7f405404ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f4061545640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405404cf70 0x7f405404ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42016/0 (socket says 172.21.15.150:42016) 2024-04-25T14:02:33.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f4061545640 1 -- 172.21.15.150:0/3316445364 learned_addr learned my addr 172.21.15.150:0/3316445364 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4054048300 0x7f40540e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.878+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff06806d6f0 con 0x7ff060048300 2024-04-25T14:02:33.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff06806dae0 con 0x7ff060048300 2024-04-25T14:02:33.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7ff0527fc640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff060005570 con 0x7ff060048300 2024-04-25T14:02:33.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.879+0000 7ff8f3fff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8ec0e08b0 0x7ff8ec0479e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4066310 tx=0x7ff8f4069ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 -- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16400340c0 msgr2=0x7f16400d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16400340c0 0x7f16400d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 -- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16400e08b0 msgr2=0x7f1640169610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16400e08b0 0x7f1640169610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164016ae40 con 0x7f16400dfd90 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3c67fff640 1 -- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c680b36e0 msgr2=0x7f3c6815a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3c67fff640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c680b36e0 0x7f3c6815a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3c67fff640 1 -- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c681576d0 msgr2=0x7f3c68202ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3c67fff640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c681576d0 0x7f3c68202ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3c67fff640 1 -- 172.21.15.150:0/3610475199 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c68157eb0 con 0x7f3c6809b4a0 2024-04-25T14:02:33.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3fa0b9c640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f9c143040 0x7f3f9c0c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7fdc260c9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc180060c0 0x7fdc18053e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc2006b610 tx=0x7fdc20076180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f5d01fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf40cc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f4060d44640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f405404e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f5d01fce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40dfd90 0x7f5cf4048450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f1645d9b640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400dfd90 0x7f16400d1fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c440 tx=0x7f163c00c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 -- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f9c143040 msgr2=0x7f3f9c0c8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f9c143040 0x7f3f9c0c8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 -- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c06dac0 msgr2=0x7f3f9c15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f9c06dac0 0x7f3f9c15a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f3f9b7fe640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9c1416b0 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f5d01fce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5cf40e06e0 0x7f5cf4039760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f5d01fce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf400c450 con 0x7f5cf40dfd90 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 -- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405404cf70 msgr2=0x7f405404ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f405404cf70 0x7f405404ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 -- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 msgr2=0x7f405404e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4075760 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c67fff640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 0x7f3c68156ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c6002acb0 tx=0x7f3c60004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3f9b7fe640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f9c0cfff0 0x7f3f9c141170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0b1110 tx=0x7f3f8c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f5d00fcc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf40cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf40cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42044/0 (socket says 172.21.15.150:42044) 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 -- 172.21.15.150:0/3494559475 learned_addr learned my addr 172.21.15.150:0/3494559475 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5cf400c2d0 con 0x7f5cf40e06e0 2024-04-25T14:02:33.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d017cd640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5cf40e06e0 0x7f5cf4039760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5cf40e38c0 con 0x7f5cf40340c0 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5cfbfff640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40dfd90 0x7f5cf4048450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f405404e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.880+0000 7f405bfff640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f405404d6e0 con 0x7f4054048300 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f4060d44640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4054047db0 0x7f405404e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f405bfff640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4054048300 0x7f40540e9e80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c4d0 tx=0x7f404c00c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8f406d6f0 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8f4075900 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60041020 con 0x7f3c6809b4a0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c60008ab0 con 0x7f3c6809b4a0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c60036040 con 0x7f3c6809b4a0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c68158190 con 0x7f3c6809b4a0 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c017070 con 0x7f4054048300 2024-04-25T14:02:33.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404c0035f0 con 0x7f4054048300 2024-04-25T14:02:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f404c0039e0 con 0x7f4054048300 2024-04-25T14:02:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c00dda0 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f8c019480 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f8c019830 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 -- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5cf40e06e0 msgr2=0x7f5cf4039760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f405404d9c0 con 0x7f4054048300 2024-04-25T14:02:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c680b1b40 con 0x7f3c6809b4a0 2024-04-25T14:02:33.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5cf40e06e0 0x7f5cf4039760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 -- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40dfd90 msgr2=0x7f5cf4048450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40dfd90 0x7f5cf4048450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5d00fcc640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cf40491c0 con 0x7f5cf40340c0 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7f5cfbfff640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5cf40dfd90 0x7f5cf4048450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f405404df80 con 0x7f4054048300 2024-04-25T14:02:33.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8ec169410 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200be070 con 0x7fdc180060c0 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc200ae650 con 0x7fdc180060c0 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc200b4040 con 0x7fdc180060c0 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c017070 con 0x7f16400dfd90 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f163c0035f0 con 0x7f16400dfd90 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f163c0039c0 con 0x7f16400dfd90 2024-04-25T14:02:33.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16400489c0 con 0x7f16400dfd90 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f164016d7b0 con 0x7f16400dfd90 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7fdc2832d640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc18052c70 con 0x7fdc180060c0 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7fdc2832d640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc18053230 con 0x7fdc180060c0 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9c0cbc50 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9c0cc210 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.883+0000 7f5d00fcc640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf40cc200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5cf002ed00 tx=0x7f5cf00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.881+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff06806dc80 con 0x7ff060048300 2024-04-25T14:02:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.883+0000 7fdc2832d640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdc18053610 con 0x7fdc180060c0 2024-04-25T14:02:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.883+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f403c001030 con 0x7f4054048300 2024-04-25T14:02:33.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3c60044020 con 0x7f3c6809b4a0 2024-04-25T14:02:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3f9c050740 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7f3c64ff9640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c540745b0 0x7f3c54076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8ec1699d0 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3c600b6870 con 0x7f3c6809b4a0 2024-04-25T14:02:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3c677fe640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c540745b0 0x7f3c54076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f162c001030 con 0x7f16400dfd90 2024-04-25T14:02:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3c6e596640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f3c68071c90 0x7f3c68145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 --> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c68076a40 con 0x7f3c68071c90 2024-04-25T14:02:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3f8c010440 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f404c01a070 con 0x7f4054048300 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff064ff9640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff04c0744e0 0x7ff04c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff068111bb0 con 0x7ff060048300 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff064ff9640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7ff04c07dcb0 0x7ff04c080140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff04c080810 con 0x7ff04c07dcb0 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.882+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff0681162a0 con 0x7ff060048300 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.883+0000 7ff067fff640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff04c0744e0 0x7ff04c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7ff0677fe640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7ff04c07dcb0 0x7ff04c080140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.884+0000 7ff067fff640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff04c0744e0 0x7ff04c0769f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009630 tx=0x7ff05c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f163c01a070 con 0x7f16400dfd90 2024-04-25T14:02:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3c6cb33640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f3c68071c90 0x7f3c68145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf003c070 con 0x7f5cf40340c0 2024-04-25T14:02:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cf0046070 con 0x7f5cf40340c0 2024-04-25T14:02:33.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7ff0677fe640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7ff04c07dcb0 0x7ff04c080140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f3f98ff9640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f7c0745b0 0x7f3f7c076ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.885+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0003850 con 0x7f5cf40340c0 2024-04-25T14:02:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f40597fa640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4048074580 0x7f4048076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f1636ffd640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1628074580 0x7f1628076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f404c0949b0 con 0x7f4054048300 2024-04-25T14:02:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f40597fa640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f404807dd40 0x7f40480801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f40480808a0 con 0x7f404807dd40 2024-04-25T14:02:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f404c0992a0 con 0x7f4054048300 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3f8c095ee0 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3f98ff9640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7f3f7c07dd80 0x7f3f7c080210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f8c00ce50 con 0x7f3f7c07dd80 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3f8c098540 con 0x7f3f9c0cfff0 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== osd.6 v2:172.21.15.187:6808/3350780520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff04c080810 con 0x7ff04c07dcb0 2024-04-25T14:02:33.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f4061545640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f404807dd40 0x7f40480801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f4060d44640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4048074580 0x7f4048076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f163c094770 con 0x7f16400dfd90 2024-04-25T14:02:33.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3f9bfff640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f7c0745b0 0x7f3f7c076ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f3fa0b9c640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7f3f7c07dd80 0x7f3f7c080210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f1636ffd640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7f162807dd40 0x7f16280801d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f16280808a0 con 0x7f162807dd40 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f163c09b050 con 0x7f16400dfd90 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.886+0000 7f164659c640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1628074580 0x7f1628076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f1646d9d640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7f162807dd40 0x7f16280801d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.943 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdc200c1070 con 0x7fdc180060c0 2024-04-25T14:02:33.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cf40497e0 con 0x7f5cf40340c0 2024-04-25T14:02:33.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7fdc12ffd640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc04074580 0x7fdc04076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fdc20132f20 con 0x7fdc180060c0 2024-04-25T14:02:33.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7fdc258c8640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc04074580 0x7fdc04076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7fdc12ffd640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7fdc0407dd50 0x7fdc040801e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdc040808b0 con 0x7fdc0407dd50 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fdc20133310 con 0x7fdc180060c0 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7fdc268ca640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7fdc0407dd50 0x7fdc040801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f1646d9d640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7f162807dd40 0x7f16280801d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f164659c640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1628074580 0x7f1628076a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f163800e040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== osd.1 v2:172.21.15.150:6816/715146204 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f16280808a0 con 0x7f162807dd40 2024-04-25T14:02:33.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f3c6cb33640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f3c68071c90 0x7f3c68145ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f3c677fe640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c540745b0 0x7f3c54076ac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c6809b920 tx=0x7f3c5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.887+0000 7f3fa0b9c640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7f3f7c07dd80 0x7f3f7c080210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f4061545640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f404807dd40 0x7f40480801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7ff8ddffb640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff8ec005570 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== osd.3 v2:172.21.15.150:6800/690753092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3f8c00ce50 con 0x7f3f7c07dd80 2024-04-25T14:02:33.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f3f9bfff640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f7c0745b0 0x7f3f7c076ac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009560 tx=0x7f3f8402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f4060d44640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4048074580 0x7f4048076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4050034000 tx=0x7f405002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.888+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== osd.2 v2:172.21.15.150:6808/2809582321 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3c68076a40 con 0x7f3c68071c90 2024-04-25T14:02:33.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.889+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cf4049da0 con 0x7f5cf40340c0 2024-04-25T14:02:33.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.889+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff8f406db20 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.889+0000 7ff8dffff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8d8074530 0x7ff8d8076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff8f4111530 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.889+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== osd.5 v2:172.21.15.187:6800/3786766423 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f40480808a0 con 0x7f404807dd40 2024-04-25T14:02:33.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8f37fe640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8d8074530 0x7ff8d8076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.889+0000 7fdc258c8640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc04074580 0x7fdc04076a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdc180e2b80 tx=0x7fdc1c007dc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8dffff640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7ff8d807dcf0 0x7ff8d8080180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8d8080850 con 0x7ff8d807dcf0 2024-04-25T14:02:33.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7ff8f4118050 con 0x7ff8ec0e08b0 2024-04-25T14:02:33.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.890+0000 7ff8f37fe640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8d8074530 0x7ff8d8076a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8034000 tx=0x7ff8e802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.891+0000 7ff8f2ffd640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7ff8d807dcf0 0x7ff8d8080180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.891+0000 7fdc268ca640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7fdc0407dd50 0x7fdc040801e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.891+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5cf404a180 con 0x7f5cf40340c0 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.891+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5cf00039f0 con 0x7f5cf40340c0 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.891+0000 7f5cf97fa640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ce0074580 0x7f5ce0076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5cf00b9eb0 con 0x7f5cf40340c0 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cf97fa640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7f5ce007dd50 0x7f5ce00801e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cf0032ea0 con 0x7f5ce007dd50 2024-04-25T14:02:33.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5cf00bc510 con 0x7f5cf40340c0 2024-04-25T14:02:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== osd.0 v2:172.21.15.150:6817/960179440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdc040808b0 con 0x7fdc0407dd50 2024-04-25T14:02:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cfbfff640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ce0074580 0x7f5ce0076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.892+0000 7f5cfbfff640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ce0074580 0x7f5ce0076a90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc066010 tx=0x7f5cfc08f040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.894+0000 7f5d017cd640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7f5ce007dd50 0x7f5ce00801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:33.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.894+0000 7ff8f2ffd640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7ff8d807dcf0 0x7ff8d8080180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.895+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== osd.7 v2:172.21.15.187:6816/931266284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff8d8080850 con 0x7ff8d807dcf0 2024-04-25T14:02:33.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.895+0000 7f5d017cd640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7f5ce007dd50 0x7f5ce00801e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:33.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.896+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== osd.4 v2:172.21.15.187:6824/361202506 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5cf0032ea0 con 0x7f5ce007dd50 2024-04-25T14:02:33.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.917+0000 7ff0527fc640 1 -- 172.21.15.150:0/2143857121 --> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff060049240 con 0x7ff04c07dcb0 2024-04-25T14:02:33.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.919+0000 7ff064ff9640 1 -- 172.21.15.150:0/2143857121 <== osd.6 v2:172.21.15.187:6808/3350780520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff060049240 con 0x7ff04c07dcb0 2024-04-25T14:02:33.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7ff04c07dcb0 msgr2=0x7ff04c080140 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7ff04c07dcb0 0x7ff04c080140 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff04c0744e0 msgr2=0x7ff04c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff04c0744e0 0x7ff04c0769f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009630 tx=0x7ff05c02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff060048300 msgr2=0x7ff06004e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 --2- 172.21.15.150:0/2143857121 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff060048300 0x7ff06004e3f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff068066310 tx=0x7ff068069ba0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06ca3c640 1 -- 172.21.15.150:0/2143857121 reap_dead start 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 shutdown_connections 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.920+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 >> 172.21.15.150:0/2143857121 conn(0x7ff060054cf0 msgr2=0x7ff060006f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.921+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 shutdown_connections 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.921+0000 7ff06d23d640 1 -- 172.21.15.150:0/2143857121 wait complete. 2024-04-25T14:02:33.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.922+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 --> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3c6815fd10 con 0x7f3c68071c90 2024-04-25T14:02:33.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.922+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 --> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3f9c0c8c00 con 0x7f3f7c07dd80 2024-04-25T14:02:33.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.922+0000 7f3c64ff9640 1 -- 172.21.15.150:0/3610475199 <== osd.2 v2:172.21.15.150:6808/2809582321 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3c6815fd10 con 0x7f3c68071c90 2024-04-25T14:02:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 --> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f162c002dd0 con 0x7f162807dd40 2024-04-25T14:02:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f1636ffd640 1 -- 172.21.15.150:0/2907058320 <== osd.1 v2:172.21.15.150:6816/715146204 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f162c002dd0 con 0x7f162807dd40 2024-04-25T14:02:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f3c68071c90 msgr2=0x7f3c68145ad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f3c68071c90 0x7f3c68145ad0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.922+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 --> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f403c002dd0 con 0x7f404807dd40 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f40597fa640 1 -- 172.21.15.150:0/3316445364 <== osd.5 v2:172.21.15.187:6800/3786766423 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f403c002dd0 con 0x7f404807dd40 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7fdc2832d640 1 -- 172.21.15.150:0/2126394628 --> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdc180055a0 con 0x7fdc0407dd50 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c540745b0 msgr2=0x7f3c54076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c540745b0 0x7f3c54076ac0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3c6809b920 tx=0x7f3c5c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 msgr2=0x7f3c68156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 --2- 172.21.15.150:0/3610475199 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6809b4a0 0x7f3c68156ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3c6002acb0 tx=0x7f3c60004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:33.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6cb33640 1 -- 172.21.15.150:0/3610475199 reap_dead start 2024-04-25T14:02:33.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7fdc12ffd640 1 -- 172.21.15.150:0/2126394628 <== osd.0 v2:172.21.15.150:6817/960179440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdc180055a0 con 0x7fdc0407dd50 2024-04-25T14:02:33.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 shutdown_connections 2024-04-25T14:02:33.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 >> 172.21.15.150:0/3610475199 conn(0x7f3c680b76d0 msgr2=0x7f3c6806e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 shutdown_connections 2024-04-25T14:02:33.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3c6e596640 1 -- 172.21.15.150:0/3610475199 wait complete. 2024-04-25T14:02:33.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3f98ff9640 1 -- 172.21.15.150:0/3612083295 <== osd.3 v2:172.21.15.150:6800/690753092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3f9c0c8c00 con 0x7f3f7c07dd80 2024-04-25T14:02:33.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.923+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f404807dd40 msgr2=0x7f40480801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f404807dd40 0x7f40480801d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4048074580 msgr2=0x7f4048076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4048074580 0x7f4048076a90 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4050034000 tx=0x7f405002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:33.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7f162807dd40 msgr2=0x7f16280801d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7f162807dd40 0x7f16280801d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1628074580 msgr2=0x7f1628076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7f3f7c07dd80 msgr2=0x7f3f7c080210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3fa139d640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7f3f7c07dd80 0x7f3f7c080210 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4054048300 msgr2=0x7f40540e9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 --2- 172.21.15.150:0/3316445364 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4054048300 0x7f40540e9e80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f404c00c4d0 tx=0x7f404c00c8d0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7fdc0407dd50 msgr2=0x7fdc040801e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7fdc10ff9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7fdc0407dd50 0x7fdc040801e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f7c0745b0 msgr2=0x7f3f7c076ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f3fa139d640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f7c0745b0 0x7f3f7c076ac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3f84009560 tx=0x7f3f8402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4061545640 1 -- 172.21.15.150:0/3316445364 reap_dead start 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 shutdown_connections 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 >> 172.21.15.150:0/3316445364 conn(0x7f4054054cf0 msgr2=0x7f40540339d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1628074580 0x7f1628076a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1638000c00 tx=0x7f163800e040 comp rx=0 tx=0).stop 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400dfd90 msgr2=0x7f16400d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.924+0000 7f164759e640 1 --2- 172.21.15.150:0/2907058320 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16400dfd90 0x7f16400d1fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f163c00c440 tx=0x7f163c00c840 comp rx=0 tx=0).stop 2024-04-25T14:02:33.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f9c0cfff0 msgr2=0x7f3f9c141170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 --2- 172.21.15.150:0/3612083295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f9c0cfff0 0x7f3f9c141170 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3f9c0b1110 tx=0x7f3f8c00ee60 comp rx=0 tx=0).stop 2024-04-25T14:02:33.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f1646d9d640 1 -- 172.21.15.150:0/2907058320 reap_dead start 2024-04-25T14:02:33.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 shutdown_connections 2024-04-25T14:02:33.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa0b9c640 1 -- 172.21.15.150:0/3612083295 reap_dead start 2024-04-25T14:02:33.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 >> 172.21.15.150:0/2907058320 conn(0x7f1640054cf0 msgr2=0x7f1640005990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc04074580 msgr2=0x7fdc04076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc10ff9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc04074580 0x7fdc04076a90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdc180e2b80 tx=0x7fdc1c007dc0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 shutdown_connections 2024-04-25T14:02:33.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 shutdown_connections 2024-04-25T14:02:33.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f4062fa8640 1 -- 172.21.15.150:0/3316445364 wait complete. 2024-04-25T14:02:33.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 >> 172.21.15.150:0/3612083295 conn(0x7f3f9c073090 msgr2=0x7f3f9c06c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc180060c0 msgr2=0x7fdc18053e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc10ff9640 1 --2- 172.21.15.150:0/2126394628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc180060c0 0x7fdc18053e50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc2006b610 tx=0x7fdc20076180 comp rx=0 tx=0).stop 2024-04-25T14:02:33.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 shutdown_connections 2024-04-25T14:02:33.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc268ca640 1 -- 172.21.15.150:0/2126394628 reap_dead start 2024-04-25T14:02:33.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f164759e640 1 -- 172.21.15.150:0/2907058320 wait complete. 2024-04-25T14:02:33.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 shutdown_connections 2024-04-25T14:02:33.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7f3fa139d640 1 -- 172.21.15.150:0/3612083295 wait complete. 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.925+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 shutdown_connections 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.926+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 >> 172.21.15.150:0/2126394628 conn(0x7fdc18054d20 msgr2=0x7fdc180367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.926+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 shutdown_connections 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.926+0000 7fdc10ff9640 1 -- 172.21.15.150:0/2126394628 wait complete. 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.927+0000 7ff8ddffb640 1 -- 172.21.15.150:0/1362142257 --> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff8ec1691c0 con 0x7ff8d807dcf0 2024-04-25T14:02:33.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.927+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 --> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5cf4005570 con 0x7f5ce007dd50 2024-04-25T14:02:33.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.928+0000 7ff8dffff640 1 -- 172.21.15.150:0/1362142257 <== osd.7 v2:172.21.15.187:6816/931266284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff8ec1691c0 con 0x7ff8d807dcf0 2024-04-25T14:02:33.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.929+0000 7f5cf97fa640 1 -- 172.21.15.150:0/3494559475 <== osd.4 v2:172.21.15.187:6824/361202506 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5cf4005570 con 0x7f5ce007dd50 2024-04-25T14:02:33.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.929+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7ff8d807dcf0 msgr2=0x7ff8d8080180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.929+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7ff8d807dcf0 0x7ff8d8080180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7f5ce007dd50 msgr2=0x7f5ce00801e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7f5ce007dd50 0x7f5ce00801e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8d8074530 msgr2=0x7ff8d8076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8d8074530 0x7ff8d8076a40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff8e8034000 tx=0x7ff8e802e000 comp rx=0 tx=0).stop 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8ec0e08b0 msgr2=0x7ff8ec0479e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 --2- 172.21.15.150:0/1362142257 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8ec0e08b0 0x7ff8ec0479e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff8f4066310 tx=0x7ff8f4069ba0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f3fff640 1 -- 172.21.15.150:0/1362142257 reap_dead start 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 shutdown_connections 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 >> 172.21.15.150:0/1362142257 conn(0x7ff8ec054cf0 msgr2=0x7ff8ec0db170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ce0074580 msgr2=0x7f5ce0076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ce0074580 0x7f5ce0076a90 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5cfc066010 tx=0x7f5cfc08f040 comp rx=0 tx=0).stop 2024-04-25T14:02:33.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 msgr2=0x7f5cf40cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:33.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d01fce640 1 --2- 172.21.15.150:0/3494559475 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5cf40340c0 0x7f5cf40cc200 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5cf002ed00 tx=0x7f5cf00046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:33.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 shutdown_connections 2024-04-25T14:02:33.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7f5d017cd640 1 -- 172.21.15.150:0/3494559475 reap_dead start 2024-04-25T14:02:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.930+0000 7ff8f8e97640 1 -- 172.21.15.150:0/1362142257 wait complete. 2024-04-25T14:02:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.931+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 shutdown_connections 2024-04-25T14:02:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.931+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 >> 172.21.15.150:0/3494559475 conn(0x7f5cf4054cf0 msgr2=0x7f5cf40dd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.931+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 shutdown_connections 2024-04-25T14:02:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:33.931+0000 7f5d01fce640 1 -- 172.21.15.150:0/3494559475 wait complete. 2024-04-25T14:02:33.977 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:33.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T14:02:33.978 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:33.978 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-04-25T14:02:33.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T14:02:33.979 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T14:02:33.980 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:33.980 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-04-25T14:02:33.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T14:02:33.981 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T14:02:33.982 INFO:teuthology.orchestra.run.smithi150.stdout:55834574861 2024-04-25T14:02:33.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T14:02:33.983 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:33.984 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T14:02:33.985 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:33.985 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T14:02:34.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.240+0000 7fc89ffff640 1 Processor -- start 2024-04-25T14:02:34.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.241+0000 7fc89ffff640 1 -- start start 2024-04-25T14:02:34.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.241+0000 7fcf6475f640 1 Processor -- start 2024-04-25T14:02:34.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.241+0000 7fcf6475f640 1 -- start start 2024-04-25T14:02:34.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7fb6a359e640 1 Processor -- start 2024-04-25T14:02:34.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7fb6a359e640 1 -- start start 2024-04-25T14:02:34.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7f7750f09640 1 Processor -- start 2024-04-25T14:02:34.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7fb8d9a5b640 1 Processor -- start 2024-04-25T14:02:34.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7f16f6492640 1 Processor -- start 2024-04-25T14:02:34.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7f16f6492640 1 -- start start 2024-04-25T14:02:34.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7f7750f09640 1 -- start start 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fb7224e4640 1 Processor -- start 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fc89ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc898047db0 0x7fc8980e7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898034060 0x7fc8980065e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fc89ffff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc898006130 con 0x7fc898048300 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fc89ffff640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc89804f690 con 0x7fc898047db0 2024-04-25T14:02:34.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fc89ffff640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc89804ffa0 con 0x7fc898034060 2024-04-25T14:02:34.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fb7224e4640 1 -- start start 2024-04-25T14:02:34.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7fcf6475f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c0b5210 0x7fcf5c0b5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf6475f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c15e300 con 0x7fcf5c15def0 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf6475f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5c0b5b60 con 0x7fcf5c0ce310 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf6475f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c0d2100 con 0x7fcf5c0b5210 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf61cfa640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcf5c15def0 0x7fcf5c0b6ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55566/0 (socket says 172.21.15.150:55566) 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf62cfc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c0b5210 0x7fcf5c0b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fcf61cfa640 1 -- 172.21.15.150:0/242427283 learned_addr learned my addr 172.21.15.150:0/242427283 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:34.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fb6a359e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb69c052420 0x7fb69c035fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fb6a359e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb69c055f60 0x7fb69c049b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fb6a359e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb69c051f70 con 0x7fb69c04a040 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fb6a359e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb69c03a1d0 con 0x7fb69c055f60 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fb6a359e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb69c056a50 con 0x7fb69c052420 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb6a2d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb69c04a040 0x7fb69c0e8210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55576/0 (socket says 172.21.15.150:55576) 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb6a2d9d640 1 -- 172.21.15.150:0/1000900188 learned_addr learned my addr 172.21.15.150:0/1000900188 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.242+0000 7fb8d9a5b640 1 -- start start 2024-04-25T14:02:34.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d9a5b640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7f7750f09640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7744053160 0x7f7744053920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7f7750f09640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f7750f09640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f7744006090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f7750f09640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7744039880 con 0x7f7744007800 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f7750f09640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7744038820 con 0x7f7744053160 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f7750f09640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7744038ac0 con 0x7f774403c4e0 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f774affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f774b7fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7744053160 0x7f7744053920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3710579757 0 0) 0x7fcf5c15e300 con 0x7fcf5c15def0 2024-04-25T14:02:34.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d8a59640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fb8cc0e0ae0 0x7fb8cc054840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:33174/0 (socket says 172.21.15.150:33174) 2024-04-25T14:02:34.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d925a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb8cc051170 0x7fb8cc0083c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55578/0 (socket says 172.21.15.150:55578) 2024-04-25T14:02:34.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf48003520 con 0x7fcf5c15def0 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330674612 0 0) 0x7fcf4c001040 con 0x7fcf5c0ce310 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 0x7f16e8038dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 0x7f16e8056070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38418/0 (socket says 172.21.15.150:38418) 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16effff640 1 -- 172.21.15.150:0/2290984171 learned_addr learned my addr 172.21.15.150:0/2290984171 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8052fa0 con 0x7f16e8053430 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f16e8039310 con 0x7f16e8055c60 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d3fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d3fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42066/0 (socket says 172.21.15.150:42066) 2024-04-25T14:02:34.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16e80520f0 con 0x7f16e80389c0 2024-04-25T14:02:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf5c15e300 con 0x7fcf5c0ce310 2024-04-25T14:02:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16f4a2f640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 0x7f16e8056070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/2290984171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 0x7f16e8038dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d9a5b640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8cc051580 con 0x7fb8cc051170 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d9a5b640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8cc0558c0 con 0x7fb8cc0e0ae0 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d9a5b640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc055020 con 0x7fb8cc0de070 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb8d8a59640 1 -- 172.21.15.150:0/3818731558 learned_addr learned my addr 172.21.15.150:0/3818731558 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f774affd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38426/0 (socket says 172.21.15.150:38426) 2024-04-25T14:02:34.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb7224e4640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb7224e4640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb71c0ce510 0x7fb71c0ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb7224e4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb71c0d31e0 con 0x7fb71c0cf520 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb7224e4640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb71c0b4590 con 0x7fb71c0ce510 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb7224e4640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c0b4f60 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7f774affd640 1 -- 172.21.15.150:0/640091494 learned_addr learned my addr 172.21.15.150:0/640091494 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 170757494 0 0) 0x7fcf48003520 con 0x7fcf5c15def0 2024-04-25T14:02:34.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c0b5210 msgr2=0x7fcf5c0b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb6a159a640 1 -- 172.21.15.150:0/1000900188 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1664053440 0 0) 0x7fb69c051f70 con 0x7fb69c04a040 2024-04-25T14:02:34.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c0b5210 0x7fcf5c0b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb7214e2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb6a159a640 1 -- 172.21.15.150:0/1000900188 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68c003520 con 0x7fb69c04a040 2024-04-25T14:02:34.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fb6a259c640 1 --2- 172.21.15.150:0/1000900188 >> v2:172.21.15.187:3300/0 conn(0x7fb69c052420 0x7fb69c035fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 >> v1:172.21.15.187:6789/0 conn(0x7fcf5c0ce310 legacy=0x7fcf5c1619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb7214e2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42094/0 (socket says 172.21.15.150:42094) 2024-04-25T14:02:34.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c162200 con 0x7fcf5c15def0 2024-04-25T14:02:34.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb7214e2640 1 -- 172.21.15.150:0/1611538009 learned_addr learned my addr 172.21.15.150:0/1611538009 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fcf58001670 con 0x7fcf5c15def0 2024-04-25T14:02:34.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16ef7fe640 1 -- 172.21.15.150:0/2290984171 >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 msgr2=0x7f16e8056070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 >> v1:172.21.15.150:6789/0 conn(0x7fcf5c15def0 legacy=0x7fcf5c0b6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f774bfff640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f7744006090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf48003bc0 0x7fcf48004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf48004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 0x7f16e8056070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb6a1d9b640 1 --2- 172.21.15.150:0/1000900188 >> v2:172.21.15.150:3300/0 conn(0x7fb69c055f60 0x7fb69c049b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7f2ebd5ef640 1 Processor -- start 2024-04-25T14:02:34.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.243+0000 7f2ebd5ef640 1 -- start start 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf480052c0 0x7fcf4800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f2ebd5ef640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2eb815d270 0x7f2eb81530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f2ebd5ef640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2eb80d0ea0 0x7f2eb806d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f2ebd5ef640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb806ded0 con 0x7f2eb806dac0 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f2ebd5ef640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb806f200 con 0x7f2eb815d270 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f2ebd5ef640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb806f4a0 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb6a259c640 1 -- 172.21.15.150:0/1000900188 >> v2:172.21.15.150:3300/0 conn(0x7fb69c055f60 msgr2=0x7fb69c049b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c0d2100 con 0x7fcf48003bc0 2024-04-25T14:02:34.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb6a259c640 1 --2- 172.21.15.150:0/1000900188 >> v2:172.21.15.150:3300/0 conn(0x7fb69c055f60 0x7fb69c049b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf4800c1e0 con 0x7fcf48004800 2024-04-25T14:02:34.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4800c3e0 con 0x7fcf480052c0 2024-04-25T14:02:34.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb6a259c640 1 -- 172.21.15.150:0/1000900188 >> v1:172.21.15.150:6789/0 conn(0x7fb69c04a040 legacy=0x7fb69c0e8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16ef7fe640 1 -- 172.21.15.150:0/2290984171 >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 msgr2=0x7f16e80e3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fb6a259c640 1 -- 172.21.15.150:0/1000900188 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c056370 con 0x7fb69c052420 2024-04-25T14:02:34.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf48004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16ef7fe640 1 -- 172.21.15.150:0/2290984171 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e80565b0 con 0x7f16e80389c0 2024-04-25T14:02:34.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f774b7fe640 1 -- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 msgr2=0x7f7744006090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f774b7fe640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f7744006090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f774b7fe640 1 -- 172.21.15.150:0/640091494 >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 msgr2=0x7f7744053e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16f4a2f640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 0x7f16e8056070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:34.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7f16effff640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:34.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.246+0000 7fcf624fb640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf48003bc0 0x7fcf48004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb8d3fff640 1 -- 172.21.15.150:0/3818731558 >> v1:172.21.15.187:6789/0 conn(0x7fb8cc0e0ae0 legacy=0x7fb8cc054840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb8d3fff640 1 -- 172.21.15.150:0/3818731558 >> v1:172.21.15.150:6789/0 conn(0x7fb8cc051170 legacy=0x7fb8cc0083c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f774b7fe640 1 --2- 172.21.15.150:0/640091494 >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f774b7fe640 1 -- 172.21.15.150:0/640091494 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7744038da0 con 0x7f7744053160 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f774affd640 1 --2- 172.21.15.150:0/640091494 >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:34.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb721ce3640 1 --2- 172.21.15.150:0/1611538009 >> v2:172.21.15.187:3300/0 conn(0x7fb71c0ce510 0x7fb71c0ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2ebcdee640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2eb806dac0 0x7f2eb815fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55592/0 (socket says 172.21.15.150:55592) 2024-04-25T14:02:34.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2ebcdee640 1 -- 172.21.15.150:0/1875583082 learned_addr learned my addr 172.21.15.150:0/1875583082 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:34.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/1875583082 >> v2:172.21.15.187:3300/0 conn(0x7f2eb80d0ea0 0x7f2eb806d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fcf61cfa640 1 -- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf480052c0 msgr2=0x7fcf4800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc898047db0 0x7fc8980e7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc898047db0 0x7fc8980e7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38410/0 (socket says 172.21.15.150:38410) 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 -- 172.21.15.150:0/1086208633 learned_addr learned my addr 172.21.15.150:0/1086208633 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 -- 172.21.15.150:0/1086208633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898034060 msgr2=0x7fc8980065e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 --2- 172.21.15.150:0/1086208633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898034060 0x7fc8980065e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.244+0000 7fc89effd640 1 -- 172.21.15.150:0/1086208633 >> v1:172.21.15.150:6789/0 conn(0x7fc898048300 legacy=0x7fc898033950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T14:02:34.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.245+0000 7fc89effd640 1 -- 172.21.15.150:0/1086208633 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc898050280 con 0x7fc898047db0 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf480052c0 0x7fcf4800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fcf61cfa640 1 -- 172.21.15.150:0/242427283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf48003bc0 msgr2=0x7fcf48004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf48003bc0 0x7fcf48004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fcf61cfa640 1 -- 172.21.15.150:0/242427283 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf48003520 con 0x7fcf48004800 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb7214e2640 1 -- 172.21.15.150:0/1611538009 >> v2:172.21.15.187:3300/0 conn(0x7fb71c0ce510 msgr2=0x7fb71c0ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb7214e2640 1 --2- 172.21.15.150:0/1611538009 >> v2:172.21.15.187:3300/0 conn(0x7fb71c0ce510 0x7fb71c0ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb713fff640 1 -- 172.21.15.150:0/1611538009 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2334391142 0 0) 0x7fb70c001110 con 0x7fb71c0cf520 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb7214e2640 1 -- 172.21.15.150:0/1611538009 >> v1:172.21.15.150:6789/0 conn(0x7fb71c0cf520 legacy=0x7fb71c0cf930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb7214e2640 1 -- 172.21.15.150:0/1611538009 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c0b3980 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fb8d3fff640 1 -- 172.21.15.150:0/3818731558 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc055300 con 0x7fb8cc0de070 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/2290984171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 0x7f16e8038dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f16e8057f50 tx=0x7f16dc02a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be6d96cb9fdacb7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7fc89effd640 1 --2- 172.21.15.150:0/1086208633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc898047db0 0x7fc8980e7d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc89800d9b0 tx=0x7fc88802a470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52804fc081e7ea5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fc89dffb640 1 -- 172.21.15.150:0/1086208633 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888036070 con 0x7fc898047db0 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fc89dffb640 1 -- 172.21.15.150:0/1086208633 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc888031600 con 0x7fc898047db0 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fc89dffb640 1 -- 172.21.15.150:0/1086208633 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888031a10 con 0x7fc898047db0 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb7fff640 1 --2- 172.21.15.150:0/1875583082 >> v2:172.21.15.150:3300/0 conn(0x7f2eb815d270 0x7f2eb81530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb77fe640 1 -- 172.21.15.150:0/1875583082 >> v2:172.21.15.150:3300/0 conn(0x7f2eb815d270 msgr2=0x7f2eb81530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/1875583082 >> v2:172.21.15.150:3300/0 conn(0x7f2eb815d270 0x7f2eb81530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb77fe640 1 -- 172.21.15.150:0/1875583082 >> v1:172.21.15.150:6789/0 conn(0x7f2eb806dac0 legacy=0x7f2eb815fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.247+0000 7f2eb77fe640 1 -- 172.21.15.150:0/1875583082 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb806f780 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f774b7fe640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7744053160 0x7f7744053920 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7744037cf0 tx=0x7f773802a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7f90fd5013b9bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fb8d3fff640 1 --2- 172.21.15.150:0/3818731558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009750 tx=0x7fb8c002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ca251299675bc3b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f16eeffd640 1 -- 172.21.15.150:0/2290984171 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc037070 con 0x7f16e80389c0 2024-04-25T14:02:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f16eeffd640 1 -- 172.21.15.150:0/2290984171 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16dc02ad90 con 0x7f16e80389c0 2024-04-25T14:02:34.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f16eeffd640 1 -- 172.21.15.150:0/2290984171 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc032660 con 0x7f16e80389c0 2024-04-25T14:02:34.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fb8d37fe640 1 -- 172.21.15.150:0/3818731558 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0037070 con 0x7fb8cc0de070 2024-04-25T14:02:34.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f774a7fc640 1 -- 172.21.15.150:0/640091494 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738037070 con 0x7f7744053160 2024-04-25T14:02:34.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f774a7fc640 1 -- 172.21.15.150:0/640091494 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773802adb0 con 0x7f7744053160 2024-04-25T14:02:34.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7f774a7fc640 1 -- 172.21.15.150:0/640091494 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738032660 con 0x7f7744053160 2024-04-25T14:02:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fb8d37fe640 1 -- 172.21.15.150:0/3818731558 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c002adb0 con 0x7fb8cc0de070 2024-04-25T14:02:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fb8d37fe640 1 -- 172.21.15.150:0/3818731558 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0032660 con 0x7fb8cc0de070 2024-04-25T14:02:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf48004c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fb0 tx=0x7fcf48005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7fd052988cb7a98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fb6a259c640 1 --2- 172.21.15.150:0/1000900188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c052420 0x7fb69c035fd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00fd00 tx=0x7fb69802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1fab853fecfac28 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58007c00 con 0x7fcf48004800 2024-04-25T14:02:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.248+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf58005ad0 con 0x7fcf48004800 2024-04-25T14:02:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7fcf614f9640 1 -- 172.21.15.150:0/242427283 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58007070 con 0x7fcf48004800 2024-04-25T14:02:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 msgr2=0x7f16e8038dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 --2- 172.21.15.150:0/2290984171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 0x7f16e8038dd0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f16e8057f50 tx=0x7f16dc02a040 comp rx=0 tx=0).stop 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 shutdown_connections 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.187:3300/0 conn(0x7f16e8055c60 0x7f16e8056070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 --2- 172.21.15.150:0/2290984171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e80389c0 0x7f16e8038dd0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 --2- 172.21.15.150:0/2290984171 >> v2:172.21.15.150:3300/0 conn(0x7f16e8053430 0x7f16e80e3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 >> 172.21.15.150:0/2290984171 conn(0x7f16e8053980 msgr2=0x7f16e8053d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/1875583082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb806d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80d07f0 tx=0x7f2ea802e9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d21b2a8622a044bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7fb7214e2640 1 --2- 172.21.15.150:0/1611538009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb714009ea0 tx=0x7fb71402a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40c7a659e3d72b6a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 -- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7744053160 msgr2=0x7f7744053920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7744053160 0x7f7744053920 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7744037cf0 tx=0x7f773802a060 comp rx=0 tx=0).stop 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 -- 172.21.15.150:0/640091494 shutdown_connections 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f7744006090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 --2- 172.21.15.150:0/640091494 >> v2:172.21.15.150:3300/0 conn(0x7f7744007800 0x7f7744053e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 --2- 172.21.15.150:0/640091494 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7744053160 0x7f7744053920 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 -- 172.21.15.150:0/640091494 >> 172.21.15.150:0/640091494 conn(0x7f7744020270 msgr2=0x7f774403bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 shutdown_connections 2024-04-25T14:02:34.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f7750f09640 1 -- 172.21.15.150:0/640091494 shutdown_connections 2024-04-25T14:02:34.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7f16f6492640 1 -- 172.21.15.150:0/2290984171 wait complete. 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7fcf6475f640 1 -- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 msgr2=0x7fcf48004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.249+0000 7fcf6475f640 1 --2- 172.21.15.150:0/242427283 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf48004c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fb0 tx=0x7fcf48005830 comp rx=0 tx=0).stop 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fcf62cfc640 1 -- 172.21.15.150:0/242427283 reap_dead start 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb713fff640 1 -- 172.21.15.150:0/1611538009 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714036070 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb713fff640 1 -- 172.21.15.150:0/1611538009 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb714031600 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb713fff640 1 -- 172.21.15.150:0/1611538009 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7140319d0 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/3818731558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 msgr2=0x7fb8cc050c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/3818731558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009750 tx=0x7fb8c002a0a0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7f7750f09640 1 -- 172.21.15.150:0/640091494 wait complete. 2024-04-25T14:02:34.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb6a159a640 1 -- 172.21.15.150:0/1000900188 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698037070 con 0x7fb69c052420 2024-04-25T14:02:34.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fcf6475f640 1 -- 172.21.15.150:0/242427283 shutdown_connections 2024-04-25T14:02:34.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fcf6475f640 1 -- 172.21.15.150:0/242427283 >> 172.21.15.150:0/242427283 conn(0x7fcf5c0d0210 msgr2=0x7fcf5c0d0620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb6a159a640 1 -- 172.21.15.150:0/1000900188 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb698032440 con 0x7fb69c052420 2024-04-25T14:02:34.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fcf6475f640 1 -- 172.21.15.150:0/242427283 shutdown_connections 2024-04-25T14:02:34.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7f2eb6ffd640 1 -- 172.21.15.150:0/1875583082 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8039020 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7f2eb6ffd640 1 -- 172.21.15.150:0/1875583082 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea8035850 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7f2eb6ffd640 1 -- 172.21.15.150:0/1875583082 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8035c20 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fcf6475f640 1 -- 172.21.15.150:0/242427283 wait complete. 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/3818731558 shutdown_connections 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.250+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/3818731558 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc050c30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/3818731558 >> 172.21.15.150:0/3818731558 conn(0x7fb8cc055e80 msgr2=0x7fb8cc056290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 -- 172.21.15.150:0/1611538009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 msgr2=0x7fb71c0b4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 --2- 172.21.15.150:0/1611538009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb714009ea0 tx=0x7fb71402a7b0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 -- 172.21.15.150:0/1086208633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc898047db0 msgr2=0x7fc8980e7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 --2- 172.21.15.150:0/1086208633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc898047db0 0x7fc8980e7d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc89800d9b0 tx=0x7fc88802a470 comp rx=0 tx=0).stop 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 -- 172.21.15.150:0/1611538009 shutdown_connections 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 --2- 172.21.15.150:0/1611538009 >> v2:172.21.15.187:3300/0 conn(0x7fb71c0ce510 0x7fb71c0ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 --2- 172.21.15.150:0/1611538009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c0b4a20 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 -- 172.21.15.150:0/1611538009 >> 172.21.15.150:0/1611538009 conn(0x7fb71c09b4a0 msgr2=0x7fb71c09b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 -- 172.21.15.150:0/1611538009 shutdown_connections 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a159a640 1 -- 172.21.15.150:0/1000900188 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb698032810 con 0x7fb69c052420 2024-04-25T14:02:34.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 -- 172.21.15.150:0/1000900188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c052420 msgr2=0x7fb69c035fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 --2- 172.21.15.150:0/1000900188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c052420 0x7fb69c035fd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00fd00 tx=0x7fb69802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb7224e4640 1 -- 172.21.15.150:0/1611538009 wait complete. 2024-04-25T14:02:34.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 -- 172.21.15.150:0/1086208633 shutdown_connections 2024-04-25T14:02:34.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/3818731558 shutdown_connections 2024-04-25T14:02:34.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 -- 172.21.15.150:0/1000900188 shutdown_connections 2024-04-25T14:02:34.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 --2- 172.21.15.150:0/1000900188 >> v2:172.21.15.150:3300/0 conn(0x7fb69c055f60 0x7fb69c049b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 --2- 172.21.15.150:0/1000900188 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c052420 0x7fb69c035fd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb6a359e640 1 -- 172.21.15.150:0/1000900188 >> 172.21.15.150:0/1000900188 conn(0x7fb69c00c3f0 msgr2=0x7fb69c00c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 --2- 172.21.15.150:0/1086208633 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898034060 0x7fc8980065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 --2- 172.21.15.150:0/1086208633 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc898047db0 0x7fc8980e7d50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fc89ffff640 1 -- 172.21.15.150:0/1086208633 >> 172.21.15.150:0/1086208633 conn(0x7fc898054cf0 msgr2=0x7fc898055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.251+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/3818731558 wait complete. 2024-04-25T14:02:34.285 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7f16f6492640 1 Processor -- start 2024-04-25T14:02:34.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7fb6a359e640 1 -- 172.21.15.150:0/1000900188 shutdown_connections 2024-04-25T14:02:34.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7fc89ffff640 1 -- 172.21.15.150:0/1086208633 shutdown_connections 2024-04-25T14:02:34.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/1875583082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 msgr2=0x7f2eb806d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/1875583082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb806d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2eb80d07f0 tx=0x7f2ea802e9e0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7f7750f09640 1 Processor -- start 2024-04-25T14:02:34.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7f7750f09640 1 -- start start 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.253+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/1875583082 shutdown_connections 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.253+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/1875583082 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb806d580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.253+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/1875583082 >> v2:172.21.15.150:3300/0 conn(0x7f2eb815d270 0x7f2eb81530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.253+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/1875583082 >> 172.21.15.150:0/1875583082 conn(0x7f2eb8073090 msgr2=0x7f2eb80734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.253+0000 7fcf6475f640 1 Processor -- start 2024-04-25T14:02:34.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7fb6a359e640 1 -- 172.21.15.150:0/1000900188 wait complete. 2024-04-25T14:02:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7fb7224e4640 1 Processor -- start 2024-04-25T14:02:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7f16f6492640 1 -- start start 2024-04-25T14:02:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7fb8d9a5b640 1 Processor -- start 2024-04-25T14:02:34.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7fb7224e4640 1 -- start start 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7f7750f09640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f774b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f774b7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38442/0 (socket says 172.21.15.150:38442) 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f774b7fe640 1 -- 172.21.15.150:0/2338542007 learned_addr learned my addr 172.21.15.150:0/2338542007 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f774404bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f774404c420 0x7f7744050630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77440e5350 con 0x7f7744007800 2024-04-25T14:02:34.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7fb6a359e640 1 Processor -- start 2024-04-25T14:02:34.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7744039360 con 0x7f774404c420 2024-04-25T14:02:34.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f774bfff640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f774404c420 0x7f7744050630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.252+0000 7fc89ffff640 1 -- 172.21.15.150:0/1086208633 wait complete. 2024-04-25T14:02:34.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7fc89ffff640 1 Processor -- start 2024-04-25T14:02:34.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7fcf6475f640 1 -- start start 2024-04-25T14:02:34.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7fc89ffff640 1 -- start start 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f77440e54d0 con 0x7f774403c4e0 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.255+0000 7f774affd640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f774404bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f774b7fe640 1 -- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 msgr2=0x7f774404bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f774b7fe640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f774404bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f774b7fe640 1 -- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f774404c420 msgr2=0x7f7744050630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f774b7fe640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f774404c420 0x7f7744050630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f774b7fe640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77440e1e20 con 0x7f7744007800 2024-04-25T14:02:34.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7fb8d9a5b640 1 -- start start 2024-04-25T14:02:34.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16e8053430 0x7f16e80d8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e8055c60 0x7f16e80cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8052950 con 0x7f16e80389c0 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f16e8052670 con 0x7f16e8053430 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16f6492640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16e8052c30 con 0x7f16e8055c60 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38452/0 (socket says 172.21.15.150:38452) 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 -- 172.21.15.150:0/774205566 learned_addr learned my addr 172.21.15.150:0/774205566 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 -- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e8055c60 msgr2=0x7f16e80cdec0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e8055c60 0x7f16e80cdec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 -- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16e8053430 msgr2=0x7f16e80d8d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7f16effff640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16e8053430 0x7f16e80d8d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7f16effff640 1 -- 172.21.15.150:0/774205566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e80d5c90 con 0x7f16e80389c0 2024-04-25T14:02:34.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/1875583082 shutdown_connections 2024-04-25T14:02:34.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.254+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/1875583082 wait complete. 2024-04-25T14:02:34.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7f2ebd5ef640 1 Processor -- start 2024-04-25T14:02:34.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc03a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb71c0ce510 0x7fb71c14cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8cc0e0ae0 0x7fb8cc03aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.256+0000 7fb6a359e640 1 -- start start 2024-04-25T14:02:34.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc0de480 con 0x7fb8cc051170 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc007ea0 con 0x7fb8cc0e0ae0 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d9a5b640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc008ad0 con 0x7fb8cc0de070 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7f16effff640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e930 tx=0x7f16e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38466/0 (socket says 172.21.15.150:38466) 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 -- 172.21.15.150:0/2655384849 learned_addr learned my addr 172.21.15.150:0/2655384849 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d925a640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8cc0e0ae0 0x7fb8cc03aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c156f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0b4220 con 0x7fb71c0ce510 2024-04-25T14:02:34.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb71c0b3f40 con 0x7fb71c0cf520 2024-04-25T14:02:34.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb7224e4640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb71c0cee60 con 0x7fb71c0d2dd0 2024-04-25T14:02:34.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d3fff640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc03a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb720ce1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb721ce3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c156f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 -- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 msgr2=0x7fb8cc03a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc03a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb720ce1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43332/0 (socket says 172.21.15.150:43332) 2024-04-25T14:02:34.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 -- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8cc0e0ae0 msgr2=0x7fb8cc03aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8cc0e0ae0 0x7fb8cc03aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7f774b7fe640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7738000c00 tx=0x7f77380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf5c203350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 0x7fcf5c06e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c0b72b0 con 0x7fcf5c206270 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5c0d1d90 con 0x7fcf48004800 2024-04-25T14:02:34.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb8d8a59640 1 -- 172.21.15.150:0/2655384849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc0361a0 con 0x7fb8cc051170 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf6475f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5c1627c0 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf61cfa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf61cfa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42120/0 (socket says 172.21.15.150:42120) 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fcf61cfa640 1 -- 172.21.15.150:0/3225638329 learned_addr learned my addr 172.21.15.150:0/3225638329 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7fb720ce1640 1 -- 172.21.15.150:0/4265665801 learned_addr learned my addr 172.21.15.150:0/4265665801 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb7214e2640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb71c0ce510 0x7fb71c14cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 -- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 msgr2=0x7fb71c156f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c156f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 -- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb71c0ce510 msgr2=0x7fb71c14cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb71c0ce510 0x7fb71c14cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 -- 172.21.15.150:0/4265665801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb71c157450 con 0x7fb71c0cf520 2024-04-25T14:02:34.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898048300 0x7fc89804e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc89817d570 0x7fc89817d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c04a040 0x7fb69c0db780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e4028070 con 0x7f16e80389c0 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf62cfc640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 0x7fcf5c06e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 -- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 msgr2=0x7fcf5c203350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf5c203350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 -- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 msgr2=0x7fcf5c06e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 0x7fcf5c06e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 -- 172.21.15.150:0/3225638329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5c15fb90 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb8d8a59640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800c550 tx=0x7fb8c800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb720ce1640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c3c0 tx=0x7fb70c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb69c052420 0x7fb69c0db000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c0536d0 con 0x7fb69c04a040 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb69c0533f0 con 0x7fb69c055f60 2024-04-25T14:02:34.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fb6a359e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb69c04a450 con 0x7fb69c052420 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf62cfc640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 0x7fcf5c06e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89effd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89effd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43340/0 (socket says 172.21.15.150:43340) 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89effd640 1 -- 172.21.15.150:0/2572508173 learned_addr learned my addr 172.21.15.150:0/2572508173 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8980509c0 con 0x7fc89817d570 2024-04-25T14:02:34.308 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc898050840 con 0x7fc898047db0 2024-04-25T14:02:34.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e400ca00 con 0x7f16e80389c0 2024-04-25T14:02:34.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7f16f6492640 1 -- 172.21.15.150:0/774205566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e80d0b30 con 0x7f16e80389c0 2024-04-25T14:02:34.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc898006b20 con 0x7fc898048300 2024-04-25T14:02:34.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.258+0000 7fcf61cfa640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fb0 tx=0x7fcf5c0d01a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f16f6492640 1 -- 172.21.15.150:0/774205566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e803b3a0 con 0x7f16e80389c0 2024-04-25T14:02:34.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89e7fc640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898048300 0x7fc89804e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.257+0000 7f2ebd5ef640 1 -- start start 2024-04-25T14:02:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e400cdb0 con 0x7f16e80389c0 2024-04-25T14:02:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43350/0 (socket says 172.21.15.150:43350) 2024-04-25T14:02:34.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 -- 172.21.15.150:0/3757177422 learned_addr learned my addr 172.21.15.150:0/3757177422 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58040020 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf58001470 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf5c15fe70 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf5c160430 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58037040 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89effd640 1 -- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898048300 msgr2=0x7fc89804e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89effd640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898048300 0x7fc89804e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89effd640 1 -- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc89817d570 msgr2=0x7fc89817d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89effd640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc89817d570 0x7fc89817d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fc89effd640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8980d9570 con 0x7fc898047db0 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 -- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb69c052420 msgr2=0x7fb69c0db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb69c052420 0x7fb69c0db000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 -- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c04a040 msgr2=0x7fb69c0db780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c04a040 0x7fb69c0db780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7fb6a2d9d640 1 -- 172.21.15.150:0/3757177422 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69c0db540 con 0x7fb69c055f60 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f2ebd5ef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 0x7f2eb8152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f2ebd5ef640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb815ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 0x7f2eb8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f2ebd5ef640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 0x7f2eb80c8170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2ebd5ef640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb8070240 con 0x7f2eb806dac0 2024-04-25T14:02:34.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb6a2d9d640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4076700 tx=0x7fb6a4069ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fc89effd640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc88802a4b0 tx=0x7fc8880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2ebd5ef640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb806fd40 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2ebd5ef640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb80d12b0 con 0x7f2eb815d270 2024-04-25T14:02:34.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a408ce40 con 0x7fb69c055f60 2024-04-25T14:02:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888041070 con 0x7fc898047db0 2024-04-25T14:02:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb6a359e640 1 -- 172.21.15.150:0/3757177422 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb69c04bc00 con 0x7fb69c055f60 2024-04-25T14:02:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2eb7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 0x7f2eb8152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38512/0 (socket says 172.21.15.150:38512) 2024-04-25T14:02:34.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc88803d460 con 0x7fc898047db0 2024-04-25T14:02:34.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb6a359e640 1 -- 172.21.15.150:0/3757177422 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb69c04c1c0 con 0x7fb69c055f60 2024-04-25T14:02:34.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc89804ee90 con 0x7fc898047db0 2024-04-25T14:02:34.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2eb7fff640 1 -- 172.21.15.150:0/3102107767 learned_addr learned my addr 172.21.15.150:0/3102107767 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:34.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc89804a4f0 con 0x7fc898047db0 2024-04-25T14:02:34.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a4069cd0 con 0x7fb69c055f60 2024-04-25T14:02:34.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a406c7b0 con 0x7fb69c055f60 2024-04-25T14:02:34.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f2ebcdee640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 0x7f2eb80c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c017070 con 0x7fb71c0cf520 2024-04-25T14:02:34.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb70c0035f0 con 0x7fb71c0cf520 2024-04-25T14:02:34.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.261+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb70c0039e0 con 0x7fb71c0cf520 2024-04-25T14:02:34.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.261+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb815ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.261+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf44005290 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738041020 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f773803d620 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7738037040 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7744054e10 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.259+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77440553d0 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.260+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7744005570 con 0x7f7744007800 2024-04-25T14:02:34.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.261+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e80059c0 con 0x7f16e80389c0 2024-04-25T14:02:34.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.262+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc888036040 con 0x7fc898047db0 2024-04-25T14:02:34.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.262+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcf58043020 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.262+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb6a406c950 con 0x7fb69c055f60 2024-04-25T14:02:34.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fcf52ffd640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 0x7fcf40076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fcf580b52c0 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc888044070 con 0x7fc898047db0 2024-04-25T14:02:34.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f16e402b070 con 0x7f16e80389c0 2024-04-25T14:02:34.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fb692ffd640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fcf624fb640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 0x7fcf40076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb6a4111860 con 0x7fb69c055f60 2024-04-25T14:02:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7fb6a259c640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7f16ecff9640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 0x7f16d4076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fc8957fa640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc884005290 con 0x7fc898047db0 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 -- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 msgr2=0x7f2eb80c8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 0x7f2eb80c8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 -- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 msgr2=0x7f2eb8152130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 0x7f2eb8152130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 -- 172.21.15.150:0/3102107767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb8152670 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb77fe640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb815ba10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8037000 tx=0x7f2ea80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2ebcdee640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 0x7f2eb80c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8044070 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea8033d50 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8039040 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.263+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb815a690 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb80c74f0 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb71c0b2410 con 0x7fb71c0cf520 2024-04-25T14:02:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb71c155ef0 con 0x7fb71c0cf520 2024-04-25T14:02:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fb6a259c640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb698034000 tx=0x7fb69802c000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fcf624fb640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 0x7fcf40076a90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0011e0 tx=0x7fcf4c0077e0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8018070 con 0x7fb8cc051170 2024-04-25T14:02:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fc8977fe640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 0x7fc880076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc8880b5d20 con 0x7fc898047db0 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.264+0000 7fc89e7fc640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 0x7fc880076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7fc89e7fc640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 0x7fc880076a90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc890006fd0 tx=0x7fc890008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8c8007500 con 0x7fb8cc051170 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c8004a20 con 0x7fb8cc051170 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7fb6a359e640 1 -- 172.21.15.150:0/3757177422 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb688005290 con 0x7fb69c055f60 2024-04-25T14:02:34.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb8050740 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f16e40a68e0 con 0x7f16e80389c0 2024-04-25T14:02:34.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 0x7f16d4076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2ea8047070 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f16ef7fe640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 0x7f16d4076a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f16dc034000 tx=0x7f16dc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.265+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7738044020 con 0x7f7744007800 2024-04-25T14:02:34.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f7737fff640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 0x7f7730076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f77380b5c40 con 0x7f7744007800 2024-04-25T14:02:34.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc035100 con 0x7fb8cc051170 2024-04-25T14:02:34.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc0356c0 con 0x7fb8cc051170 2024-04-25T14:02:34.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f774affd640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 0x7f7730076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb71c0b2160 con 0x7fb71c0cf520 2024-04-25T14:02:34.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.266+0000 7f2eb4ff9640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 0x7f2ea40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.267+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f2ea80b6d30 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.267+0000 7f2eb7fff640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 0x7f2ea40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.267+0000 7f2eb7fff640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 0x7f2ea40769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007a60 tx=0x7f2eac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.267+0000 7f774affd640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 0x7f7730076a90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f774c000b60 tx=0x7f774c08f040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.268+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8cc0059c0 con 0x7fb8cc051170 2024-04-25T14:02:34.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.270+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f16e4070d10 con 0x7f16e80389c0 2024-04-25T14:02:34.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.272+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcf5804a030 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.274+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc888080190 con 0x7fc898047db0 2024-04-25T14:02:34.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.275+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7738080070 con 0x7f7744007800 2024-04-25T14:02:34.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.275+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb70c01a070 con 0x7fb71c0cf520 2024-04-25T14:02:34.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.276+0000 7fb711ffb640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 0x7fb700076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.276+0000 7fb7214e2640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 0x7fb700076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.277+0000 7fb7214e2640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 0x7fb700076a40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb714033000 tx=0x7fb71402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.279+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb8c8004bc0 con 0x7fb8cc051170 2024-04-25T14:02:34.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.279+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb6a40dbca0 con 0x7fb69c055f60 2024-04-25T14:02:34.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.280+0000 7fb8d17fa640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 0x7fb8b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:34.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.280+0000 7fb8d3fff640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 0x7fb8b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:34.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.280+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb8c8094ad0 con 0x7fb8cc051170 2024-04-25T14:02:34.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.281+0000 7fb8d3fff640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 0x7fb8b80769c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0034000 tx=0x7fb8c002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:34.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.284+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2eb8152670 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.284+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fb70c013df0 con 0x7fb71c0cf520 2024-04-25T14:02:34.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.285+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8c805ef90 con 0x7fb8cc051170 2024-04-25T14:02:34.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.289+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb70c021020 con 0x7fb71c0cf520 2024-04-25T14:02:34.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.688+0000 7fc8957fa640 1 -- 172.21.15.150:0/2572508173 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fc884005580 con 0x7fc898047db0 2024-04-25T14:02:34.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.688+0000 7fc8977fe640 1 -- 172.21.15.150:0/2572508173 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc88807fa90 con 0x7fc898047db0 2024-04-25T14:02:34.688 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:34.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.691+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f16e8000e70 con 0x7f16e80389c0 2024-04-25T14:02:34.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 msgr2=0x7fc880076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7f16ecff9640 1 -- 172.21.15.150:0/774205566 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f16e4070610 con 0x7f16e80389c0 2024-04-25T14:02:34.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 0x7fc880076a90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc890006fd0 tx=0x7fc890008040 comp rx=0 tx=0).stop 2024-04-25T14:02:34.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 msgr2=0x7fc8980d6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc88802a4b0 tx=0x7fc8880046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 shutdown_connections 2024-04-25T14:02:34.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc880074580 0x7fc880076a90 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc890006fd0 tx=0x7fc890008040 comp rx=0 tx=0).stop 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc89817d570 0x7fc89817d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc898048300 0x7fc89804e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 --2- 172.21.15.150:0/2572508173 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc898047db0 0x7fc8980d6330 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 >> 172.21.15.150:0/2572508173 conn(0x7fc898054cf0 msgr2=0x7fc898034750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 shutdown_connections 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.692+0000 7fc89ffff640 1 -- 172.21.15.150:0/2572508173 wait complete. 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:34.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.694+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fcf44005580 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.695+0000 7fcf52ffd640 1 -- 172.21.15.150:0/3225638329 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcf58005aa0 con 0x7fcf5c15c6f0 2024-04-25T14:02:34.694 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-04-25T14:02:34.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 msgr2=0x7f16d4076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 0x7f16d4076a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f16dc034000 tx=0x7f16dc02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 msgr2=0x7f16e80d4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e930 tx=0x7f16e400edf0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 shutdown_connections 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16d4074580 0x7f16d4076a90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16e8055c60 0x7f16e80cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16e8053430 0x7f16e80d8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 --2- 172.21.15.150:0/774205566 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f16e80389c0 0x7f16e80d4ec0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f16e400e930 tx=0x7f16e400edf0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.696+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 >> 172.21.15.150:0/774205566 conn(0x7f16e8053980 msgr2=0x7f16e8037cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.697+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 shutdown_connections 2024-04-25T14:02:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.697+0000 7f16da7fc640 1 -- 172.21.15.150:0/774205566 wait complete. 2024-04-25T14:02:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.698+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 msgr2=0x7fcf40076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.698+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 0x7fcf40076a90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c0011e0 tx=0x7fcf4c0077e0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 msgr2=0x7fcf5c0cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fcf58007fb0 tx=0x7fcf5c0d01a0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 shutdown_connections 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcf40074580 0x7fcf40076a90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcf5c206270 0x7fcf5c06e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcf5c15c6f0 0x7fcf5c0cb490 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 --2- 172.21.15.150:0/3225638329 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcf48004800 0x7fcf5c203350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.699+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 >> 172.21.15.150:0/3225638329 conn(0x7fcf5c0d0210 msgr2=0x7fcf5c0b3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.700+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 shutdown_connections 2024-04-25T14:02:34.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.700+0000 7fcf6475f640 1 -- 172.21.15.150:0/3225638329 wait complete. 2024-04-25T14:02:34.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.701+0000 7fb6a359e640 1 -- 172.21.15.150:0/3757177422 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb688005580 con 0x7fb69c055f60 2024-04-25T14:02:34.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.702+0000 7fb692ffd640 1 -- 172.21.15.150:0/3757177422 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb6a40db5a0 con 0x7fb69c055f60 2024-04-25T14:02:34.701 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:34.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.705+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 msgr2=0x7fb6840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb698034000 tx=0x7fb69802c000 comp rx=0 tx=0).stop 2024-04-25T14:02:34.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb71c06f000 con 0x7fb71c0cf520 2024-04-25T14:02:34.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 msgr2=0x7fb69c1815a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb6a4076700 tx=0x7fb6a4069ba0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 shutdown_connections 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69c055f60 0x7fb69c1815a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb69c052420 0x7fb69c0db000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 --2- 172.21.15.150:0/3757177422 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb69c04a040 0x7fb69c0db780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 >> 172.21.15.150:0/3757177422 conn(0x7fb69c00c3f0 msgr2=0x7fb69c005bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 shutdown_connections 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb690ff9640 1 -- 172.21.15.150:0/3757177422 wait complete. 2024-04-25T14:02:34.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.706+0000 7fb711ffb640 1 -- 172.21.15.150:0/4265665801 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb71c0d31e0 con 0x7fb71c0cf520 2024-04-25T14:02:34.707 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-04-25T14:02:34.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.707+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f7744054730 con 0x7f7744007800 2024-04-25T14:02:34.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.708+0000 7f7737fff640 1 -- 172.21.15.150:0/2338542007 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f77380880f0 con 0x7f7744007800 2024-04-25T14:02:34.708 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:34.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.709+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2eb8070f30 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 msgr2=0x7fb700076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 0x7fb700076a40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb714033000 tx=0x7fb71402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:34.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7f2eb4ff9640 1 -- 172.21.15.150:0/3102107767 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2ea8080b20 con 0x7f2eb80d0ea0 2024-04-25T14:02:34.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 msgr2=0x7fb71c1569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb70c00c3c0 tx=0x7fb70c00c880 comp rx=0 tx=0).stop 2024-04-25T14:02:34.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 shutdown_connections 2024-04-25T14:02:34.710 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:34.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.710+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb8cc0d9950 con 0x7fb8cc051170 2024-04-25T14:02:34.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb700074530 0x7fb700076a40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb71c0d2dd0 0x7fb71c156f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb71c0cf520 0x7fb71c1569d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 --2- 172.21.15.150:0/4265665801 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb71c0ce510 0x7fb71c14cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 >> 172.21.15.150:0/4265665801 conn(0x7fb71c09b4a0 msgr2=0x7fb71c09b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 shutdown_connections 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb7224e4640 1 -- 172.21.15.150:0/4265665801 wait complete. 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.711+0000 7fb8d17fa640 1 -- 172.21.15.150:0/2655384849 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8c805e890 con 0x7fb8cc051170 2024-04-25T14:02:34.712 INFO:teuthology.orchestra.run.smithi150.stdout:55834574857 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 msgr2=0x7f7730076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 0x7f7730076a90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f774c000b60 tx=0x7f774c08f040 comp rx=0 tx=0).stop 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 msgr2=0x7f77440df430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7738000c00 tx=0x7f77380046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 shutdown_connections 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7730074580 0x7f7730076a90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f774404c420 0x7f7744050630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f774403c4e0 0x7f774404bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.713+0000 7f7750f09640 1 --2- 172.21.15.150:0/2338542007 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7744007800 0x7f77440df430 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 >> 172.21.15.150:0/2338542007 conn(0x7f7744020270 msgr2=0x7f77440e1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 shutdown_connections 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f7750f09640 1 -- 172.21.15.150:0/2338542007 wait complete. 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 msgr2=0x7f2ea40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 0x7f2ea40769c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2eac007a60 tx=0x7f2eac008040 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 msgr2=0x7f2eb815ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb815ba10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8037000 tx=0x7f2ea80046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.714+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 shutdown_connections 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2ea40744b0 0x7f2ea40769c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2eb815d270 0x7f2eb80c8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2eb80d0ea0 0x7f2eb815ba10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 --2- 172.21.15.150:0/3102107767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2eb806dac0 0x7f2eb8152130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 >> 172.21.15.150:0/3102107767 conn(0x7f2eb8073090 msgr2=0x7f2eb806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 shutdown_connections 2024-04-25T14:02:34.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.715+0000 7f2ebd5ef640 1 -- 172.21.15.150:0/3102107767 wait complete. 2024-04-25T14:02:34.716 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.0 2024-04-25T14:02:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 msgr2=0x7fb8b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 0x7fb8b80769c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0034000 tx=0x7fb8c002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 msgr2=0x7fb8cc038210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb8c800c550 tx=0x7fb8c800c950 comp rx=0 tx=0).stop 2024-04-25T14:02:34.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 shutdown_connections 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8b80744b0 0x7fb8b80769c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8cc0e0ae0 0x7fb8cc03aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8cc0de070 0x7fb8cc03a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 --2- 172.21.15.150:0/2655384849 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8cc051170 0x7fb8cc038210 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 >> 172.21.15.150:0/2655384849 conn(0x7fb8cc055e80 msgr2=0x7fb8cc0dd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 shutdown_connections 2024-04-25T14:02:34.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:34.716+0000 7fb8d9a5b640 1 -- 172.21.15.150:0/2655384849 wait complete. 2024-04-25T14:02:34.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.4 2024-04-25T14:02:34.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-04-25T14:02:34.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.6 2024-04-25T14:02:34.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.3 2024-04-25T14:02:34.736 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574860 for osd.5 2024-04-25T14:02:34.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-04-25T14:02:34.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574859 for osd.7 2024-04-25T14:02:35.717 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-25T14:02:35.721 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-25T14:02:35.723 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-25T14:02:35.729 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-25T14:02:35.731 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-25T14:02:35.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-25T14:02:35.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-25T14:02:35.738 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-25T14:02:35.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.910+0000 7f618bfff640 1 Processor -- start 2024-04-25T14:02:35.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.911+0000 7f618bfff640 1 -- start start 2024-04-25T14:02:35.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.914+0000 7f618bfff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6184053de0 0x7f61840e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.914+0000 7f618bfff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f6184033d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.914+0000 7f618bfff640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61840342a0 con 0x7f61840e0350 2024-04-25T14:02:35.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.914+0000 7f618bfff640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f61840e3850 con 0x7f6184053de0 2024-04-25T14:02:35.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.914+0000 7f618bfff640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6184050460 con 0x7f6184033950 2024-04-25T14:02:35.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.916+0000 7f618affd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6184053de0 0x7f61840e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.916+0000 7f618affd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6184053de0 0x7f61840e3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43374/0 (socket says 172.21.15.150:43374) 2024-04-25T14:02:35.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.916+0000 7f618affd640 1 -- 172.21.15.150:0/2523180234 learned_addr learned my addr 172.21.15.150:0/2523180234 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:35.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.917+0000 7f618a7fc640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f6184033d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.917+0000 7f618affd640 1 -- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 msgr2=0x7f6184033d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:35.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.918+0000 7f618affd640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f6184033d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.918+0000 7f618affd640 1 -- 172.21.15.150:0/2523180234 >> v1:172.21.15.187:6789/0 conn(0x7f61840e0350 legacy=0x7f61840e0760 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T14:02:35.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.918+0000 7f618affd640 1 -- 172.21.15.150:0/2523180234 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61840e3520 con 0x7f6184053de0 2024-04-25T14:02:35.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.918+0000 7f618a7fc640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f6184033d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:35.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.920+0000 7f618affd640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840e3ce0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f617c009870 tx=0x7f617c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b38725f38d7f583a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:35.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.920+0000 7f6189ffb640 1 -- 172.21.15.150:0/2523180234 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c037070 con 0x7f6184053de0 2024-04-25T14:02:35.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.921+0000 7f6189ffb640 1 -- 172.21.15.150:0/2523180234 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f617c032440 con 0x7f6184053de0 2024-04-25T14:02:35.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.921+0000 7f6189ffb640 1 -- 172.21.15.150:0/2523180234 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f617c032810 con 0x7f6184053de0 2024-04-25T14:02:35.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 -- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 msgr2=0x7f61840e3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840e3ce0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f617c009870 tx=0x7f617c02a180 comp rx=0 tx=0).stop 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 -- 172.21.15.150:0/2523180234 shutdown_connections 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f6184033d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 --2- 172.21.15.150:0/2523180234 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840e3ce0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 -- 172.21.15.150:0/2523180234 >> 172.21.15.150:0/2523180234 conn(0x7f6184054cf0 msgr2=0x7f6184055100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.923+0000 7f618bfff640 1 -- 172.21.15.150:0/2523180234 shutdown_connections 2024-04-25T14:02:35.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.924+0000 7f618bfff640 1 -- 172.21.15.150:0/2523180234 wait complete. 2024-04-25T14:02:35.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.927+0000 7f618bfff640 1 Processor -- start 2024-04-25T14:02:35.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.928+0000 7f618bfff640 1 -- start start 2024-04-25T14:02:35.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.930+0000 7f618bfff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f618404bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.930+0000 7f618bfff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.930+0000 7f618bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 0x7f61840defb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618bfff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f61840e4220 con 0x7f61840e0350 2024-04-25T14:02:35.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618bfff640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61840509e0 con 0x7f6184053de0 2024-04-25T14:02:35.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618a7fc640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43388/0 (socket says 172.21.15.150:43388) 2024-04-25T14:02:35.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618a7fc640 1 -- 172.21.15.150:0/2912096895 learned_addr learned my addr 172.21.15.150:0/2912096895 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:35.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.931+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6184050cc0 con 0x7f6184033950 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618b7fe640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 0x7f61840defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 -- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 msgr2=0x7f618404bfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f618404bfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 -- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 msgr2=0x7f61840defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 0x7f61840defb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6184048d00 con 0x7f6184053de0 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618b7fe640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 0x7f61840defb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:35.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.932+0000 7f618a7fc640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f618c06af00 tx=0x7f618c069ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:35.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.933+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c0756e0 con 0x7f6184053de0 2024-04-25T14:02:35.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.933+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f618404ebd0 con 0x7f6184053de0 2024-04-25T14:02:35.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.933+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6184049250 con 0x7f6184053de0 2024-04-25T14:02:35.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.935+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f618c075880 con 0x7f6184053de0 2024-04-25T14:02:35.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.935+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c06d8f0 con 0x7f6184053de0 2024-04-25T14:02:35.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.937+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f618c06db50 con 0x7f6184053de0 2024-04-25T14:02:35.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.938+0000 7f61737fe640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 0x7f6178076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.938+0000 7f618affd640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 0x7f6178076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.939+0000 7f618affd640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 0x7f6178076a90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f617c034000 tx=0x7f617c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:35.938 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.939+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f618c112980 con 0x7f6184053de0 2024-04-25T14:02:35.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.940+0000 7f61717fa640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f616c005290 con 0x7f6184053de0 2024-04-25T14:02:35.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.946+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f618c0dce00 con 0x7f6184053de0 2024-04-25T14:02:35.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.979+0000 7facca18f640 1 Processor -- start 2024-04-25T14:02:35.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.979+0000 7facca18f640 1 -- start start 2024-04-25T14:02:35.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7effbb430640 1 Processor -- start 2024-04-25T14:02:35.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7effbb430640 1 -- start start 2024-04-25T14:02:35.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7facca18f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 0x7facc40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7facca18f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc40d0200 con 0x7facc40cfdf0 2024-04-25T14:02:35.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7facca18f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facc415e140 con 0x7facc40d2f90 2024-04-25T14:02:35.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7facca18f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facc40cf310 con 0x7facc415db90 2024-04-25T14:02:35.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7f3f89849640 1 Processor -- start 2024-04-25T14:02:35.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.981+0000 7f27028de640 1 Processor -- start 2024-04-25T14:02:35.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7f27028de640 1 -- start start 2024-04-25T14:02:35.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7fd9b1dff640 1 Processor -- start 2024-04-25T14:02:35.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7fd9b1dff640 1 -- start start 2024-04-25T14:02:35.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7facc40cfdf0 0x7facc40d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55606/0 (socket says 172.21.15.150:55606) 2024-04-25T14:02:35.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc37fe640 1 -- 172.21.15.150:0/3989531984 learned_addr learned my addr 172.21.15.150:0/3989531984 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:35.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc3fff640 1 --2- 172.21.15.150:0/3989531984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 0x7facc40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7f45d96b1640 1 Processor -- start 2024-04-25T14:02:35.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc27fc640 1 -- 172.21.15.150:0/3989531984 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4104503834 0 0) 0x7facc40d0200 con 0x7facc40cfdf0 2024-04-25T14:02:35.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc3fff640 1 -- 172.21.15.150:0/3989531984 >> v1:172.21.15.187:6789/0 conn(0x7facc40d2f90 legacy=0x7facc415d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:35.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc3fff640 1 -- 172.21.15.150:0/3989531984 >> v1:172.21.15.150:6789/0 conn(0x7facc40cfdf0 legacy=0x7facc40d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:35.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7facc3fff640 1 -- 172.21.15.150:0/3989531984 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc40cf5f0 con 0x7facc415db90 2024-04-25T14:02:35.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7f45d96b1640 1 -- start start 2024-04-25T14:02:35.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.983+0000 7effbb430640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effb40d3320 0x7effb40b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.983+0000 7facc3fff640 1 --2- 172.21.15.150:0/3989531984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 0x7facc40b2200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7facbc009dc0 tx=0x7facbc02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3f5edb83dbf9b27 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:35.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.983+0000 7effb91cc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7effb41576d0 0x7effb415db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55614/0 (socket says 172.21.15.150:55614) 2024-04-25T14:02:35.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effb91cc640 1 -- 172.21.15.150:0/3205056244 learned_addr learned my addr 172.21.15.150:0/3205056244 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:35.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7facc27fc640 1 -- 172.21.15.150:0/3989531984 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc007740 con 0x7facc415db90 2024-04-25T14:02:35.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7facc27fc640 1 -- 172.21.15.150:0/3989531984 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facbc0078e0 con 0x7facc415db90 2024-04-25T14:02:35.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7facc27fc640 1 -- 172.21.15.150:0/3989531984 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facbc007cf0 con 0x7facc415db90 2024-04-25T14:02:35.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb415e3d0 con 0x7effb41576d0 2024-04-25T14:02:35.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1242397699 0 0) 0x7effb415e3d0 con 0x7effb41576d0 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.982+0000 7f3f89849640 1 -- start start 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc143120 0x7f26fc15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f3f89849640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f840cfa80 0x7f3f840c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f3f89849640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f84073f50 0x7f3f840cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f3f89849640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f840c76e0 con 0x7f3f84164a80 2024-04-25T14:02:35.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effb40b2920 con 0x7effb409b4a0 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7effb40b3250 con 0x7effb40d3320 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effa0007930 con 0x7effb41576d0 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7fd9b1dff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd9a4038c60 0x7fd9a403afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f26fc06d580 0x7f26fc06d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f26fc160860 0x7f26fc160c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f26fc0ce5b0 con 0x7f26fc06d580 2024-04-25T14:02:35.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f26fc06ded0 con 0x7f26fc160860 2024-04-25T14:02:35.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f27028de640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc163bd0 con 0x7f26fc143120 2024-04-25T14:02:35.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f3f89849640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3f840cd3d0 con 0x7f3f84073f50 2024-04-25T14:02:35.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7f3f89849640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3f840c79c0 con 0x7f3f840cfa80 2024-04-25T14:02:35.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7fd9b1dff640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a403b670 con 0x7fd9a4009590 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40d4380 0x7f45d40b4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45d406d8f0 0x7f45d40b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 0x7f45d415f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f45d406c570 con 0x7f45d40b9600 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f45d406c850 con 0x7f45d406d8f0 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d96b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45d406cb30 con 0x7f45d40d4380 2024-04-25T14:02:35.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7fd9b1dff640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a403b950 con 0x7fd9a40e1790 2024-04-25T14:02:35.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.984+0000 7fd9b1dff640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a4007150 con 0x7fd9a4038c60 2024-04-25T14:02:35.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f3f827fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3f84164a80 0x7f3f84073840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55622/0 (socket says 172.21.15.150:55622) 2024-04-25T14:02:35.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f27020dd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f26fc160860 0x7f26fc160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f3f827fc640 1 -- 172.21.15.150:0/2278533173 learned_addr learned my addr 172.21.15.150:0/2278533173 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:35.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d27fc640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45d406d8f0 0x7f45d40b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f3f837fe640 1 --2- 172.21.15.150:0/2278533173 >> v2:172.21.15.150:3300/0 conn(0x7f3f84073f50 0x7f3f840cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f27018dc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc143120 0x7f26fc15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d27fc640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f45d406d8f0 0x7f45d40b90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43404/0 (socket says 172.21.15.150:43404) 2024-04-25T14:02:35.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d27fc640 1 -- 172.21.15.150:0/924878458 learned_addr learned my addr 172.21.15.150:0/924878458 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:35.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f27020dd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f26fc160860 0x7f26fc160c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43398/0 (socket says 172.21.15.150:43398) 2024-04-25T14:02:35.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f27020dd640 1 -- 172.21.15.150:0/2047147340 learned_addr learned my addr 172.21.15.150:0/2047147340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:35.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d2ffd640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40d4380 0x7f45d40b4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f3f82ffd640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f840cfa80 0x7f3f840c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f27010db640 1 --2- 172.21.15.150:0/2047147340 >> v2:172.21.15.150:3300/0 conn(0x7f26fc06d580 0x7f26fc06d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 354291939 0 0) 0x7effa0007930 con 0x7effb41576d0 2024-04-25T14:02:35.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 >> v2:172.21.15.150:3300/0 conn(0x7effb40d3320 msgr2=0x7effb40b36e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T14:02:35.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7effabfff640 1 --2- 172.21.15.150:0/3205056244 >> v2:172.21.15.150:3300/0 conn(0x7effb40d3320 0x7effb40b36e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 >> v1:172.21.15.187:6789/0 conn(0x7effb409b4a0 legacy=0x7effb409b8b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb40b2c00 con 0x7effb41576d0 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7facca18f640 1 -- 172.21.15.150:0/3989531984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 msgr2=0x7facc40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7facca18f640 1 --2- 172.21.15.150:0/3989531984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 0x7facc40b2200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7facbc009dc0 tx=0x7facbc02a660 comp rx=0 tx=0).stop 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7facca18f640 1 -- 172.21.15.150:0/3989531984 shutdown_connections 2024-04-25T14:02:35.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7facca18f640 1 --2- 172.21.15.150:0/3989531984 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc415db90 0x7facc40b2200 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:35.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7facca18f640 1 -- 172.21.15.150:0/3989531984 >> 172.21.15.150:0/3989531984 conn(0x7facc409b4a0 msgr2=0x7facc409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:35.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7fd9ab7fe640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd9a40e1790 0x7fd9a4008e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:33218/0 (socket says 172.21.15.150:33218) 2024-04-25T14:02:35.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7fd9ab7fe640 1 -- 172.21.15.150:0/803306267 learned_addr learned my addr 172.21.15.150:0/803306267 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:35.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.985+0000 7f45d37fe640 1 --2- 172.21.15.150:0/924878458 >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 0x7f45d415f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7fd9abfff640 1 --2- 172.21.15.150:0/803306267 >> v2:172.21.15.187:3300/0 conn(0x7fd9a4038c60 0x7fd9a403afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:35.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7effac0019f0 con 0x7effb41576d0 2024-04-25T14:02:35.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f073f77a640 1 Processor -- start 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 >> v1:172.21.15.150:6789/0 conn(0x7effb41576d0 legacy=0x7effb415db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 0x7effa0008230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa0008c10 0x7effa0009020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa0009800 0x7effa000fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b3250 con 0x7effa0008c10 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7effa00107a0 con 0x7effa0009800 2024-04-25T14:02:36.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effa00109a0 con 0x7effa0007dc0 2024-04-25T14:02:36.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d27fc640 1 -- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40d4380 msgr2=0x7f45d40b4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb89cb640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa0008c10 0x7effa0009020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d27fc640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40d4380 0x7f45d40b4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d27fc640 1 -- 172.21.15.150:0/924878458 >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 msgr2=0x7f45d415f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d27fc640 1 --2- 172.21.15.150:0/924878458 >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 0x7f45d415f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb99cd640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa0009800 0x7effa000fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 0x7effa0008230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7facca18f640 1 -- 172.21.15.150:0/3989531984 shutdown_connections 2024-04-25T14:02:36.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7facca18f640 1 -- 172.21.15.150:0/3989531984 wait complete. 2024-04-25T14:02:36.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d27fc640 1 -- 172.21.15.150:0/924878458 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d406ce10 con 0x7f45d406d8f0 2024-04-25T14:02:36.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f45d37fe640 1 --2- 172.21.15.150:0/924878458 >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 0x7f45d415f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:36.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146563721 0 0) 0x7fd9a403b670 con 0x7fd9a4009590 2024-04-25T14:02:36.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd99c003520 con 0x7fd9a4009590 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 -- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa0009800 msgr2=0x7effa000fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa0009800 0x7effa000fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 -- 172.21.15.150:0/3205056244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa0008c10 msgr2=0x7effa0009020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa0008c10 0x7effa0009020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb91cc640 1 -- 172.21.15.150:0/3205056244 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa0007930 con 0x7effa0007dc0 2024-04-25T14:02:36.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7effb99cd640 1 -- 172.21.15.150:0/3205056244 reap_dead start 2024-04-25T14:02:36.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f27020dd640 1 -- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc143120 msgr2=0x7f26fc15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f27020dd640 1 --2- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc143120 0x7f26fc15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f27020dd640 1 -- 172.21.15.150:0/2047147340 >> v2:172.21.15.150:3300/0 conn(0x7f26fc06d580 msgr2=0x7f26fc06d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f27020dd640 1 --2- 172.21.15.150:0/2047147340 >> v2:172.21.15.150:3300/0 conn(0x7f26fc06d580 0x7f26fc06d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f27020dd640 1 -- 172.21.15.150:0/2047147340 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc0cd6e0 con 0x7f26fc160860 2024-04-25T14:02:36.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 779406384 0 0) 0x7fd9a403b950 con 0x7fd9a40e1790 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9a403b670 con 0x7fd9a40e1790 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.1 v1:172.21.15.187:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 886046799 0 0) 0x7fd9a403b670 con 0x7fd9a40e1790 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9abfff640 1 -- 172.21.15.150:0/803306267 >> v1:172.21.15.187:6789/0 conn(0x7fd9a40e1790 legacy=0x7fd9a4008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9abfff640 1 -- 172.21.15.150:0/803306267 >> v1:172.21.15.150:6789/0 conn(0x7fd9a4009590 legacy=0x7fd9a40078c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7fd9abfff640 1 -- 172.21.15.150:0/803306267 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a403bc30 con 0x7fd9a4038c60 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f3f837fe640 1 -- 172.21.15.150:0/2278533173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f840cfa80 msgr2=0x7f3f840c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f3f837fe640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f840cfa80 0x7f3f840c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f3f837fe640 1 -- 172.21.15.150:0/2278533173 >> v1:172.21.15.150:6789/0 conn(0x7f3f84164a80 legacy=0x7f3f84073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.986+0000 7f3f837fe640 1 -- 172.21.15.150:0/2278533173 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f840c7ca0 con 0x7f3f84073f50 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f3f837fe640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f84073f50 0x7f3f840cd860 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097a0 tx=0x7f3f7402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23127bcf98b39fc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7effb91cc640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 0x7effa0008230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effb409c590 tx=0x7effa0009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2e2b3db7794b8c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac00a4a0 con 0x7effa0007dc0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effac02f070 con 0x7effa0007dc0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7effabfff640 1 -- 172.21.15.150:0/3205056244 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac02bcb0 con 0x7effa0007dc0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7f45d27fc640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406d8f0 0x7f45d40b90c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45d40b87e0 tx=0x7f45cc032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28f8a093120c5e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7f45d1ffb640 1 -- 172.21.15.150:0/924878458 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc033a00 con 0x7f45d406d8f0 2024-04-25T14:02:36.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f45d1ffb640 1 -- 172.21.15.150:0/924878458 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc033ba0 con 0x7f45d406d8f0 2024-04-25T14:02:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f45d1ffb640 1 -- 172.21.15.150:0/924878458 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc03b3f0 con 0x7f45d406d8f0 2024-04-25T14:02:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.987+0000 7f073f77a640 1 -- start start 2024-04-25T14:02:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f3f81ffb640 1 -- 172.21.15.150:0/2278533173 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74037070 con 0x7f3f84073f50 2024-04-25T14:02:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f3f81ffb640 1 -- 172.21.15.150:0/2278533173 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f74032440 con 0x7f3f84073f50 2024-04-25T14:02:36.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f3f81ffb640 1 -- 172.21.15.150:0/2278533173 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f74037070 con 0x7f3f84073f50 2024-04-25T14:02:36.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7fd9abfff640 1 --2- 172.21.15.150:0/803306267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4038c60 0x7fd9a403afb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40072d0 tx=0x7fd9a002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5d31f6a181d1a56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f27020dd640 1 --2- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc160860 0x7f26fc160c70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0706a0 tx=0x7f26f402a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=300cd5b5f5129d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f27008da640 1 -- 172.21.15.150:0/2047147340 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4036070 con 0x7f26fc160860 2024-04-25T14:02:36.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a0037070 con 0x7fd9a4038c60 2024-04-25T14:02:36.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a002abe0 con 0x7fd9a4038c60 2024-04-25T14:02:36.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7f45d96b1640 1 -- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406d8f0 msgr2=0x7f45d40b90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7f45d96b1640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406d8f0 0x7f45d40b90c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45d40b87e0 tx=0x7f45cc032ec0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7facca18f640 1 Processor -- start 2024-04-25T14:02:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.988+0000 7f27008da640 1 -- 172.21.15.150:0/2047147340 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f4031600 con 0x7f26fc160860 2024-04-25T14:02:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7f27008da640 1 -- 172.21.15.150:0/2047147340 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4031a10 con 0x7f26fc160860 2024-04-25T14:02:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7facca18f640 1 -- start start 2024-04-25T14:02:36.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7fd9aa7fc640 1 -- 172.21.15.150:0/803306267 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998001670 con 0x7fd9a4038c60 2024-04-25T14:02:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 msgr2=0x7effa0008230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 0x7effa0008230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effb409c590 tx=0x7effa0009c40 comp rx=0 tx=0).stop 2024-04-25T14:02:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 shutdown_connections 2024-04-25T14:02:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 --2- 172.21.15.150:0/3205056244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa0007dc0 0x7effa0008230 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7effb409c590 tx=0x7effa0009c40 comp rx=0 tx=0).stop 2024-04-25T14:02:36.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 >> 172.21.15.150:0/3205056244 conn(0x7effb40b76d0 msgr2=0x7effb40b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7f073f77a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f073815f760 0x7f0738161b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.989+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 shutdown_connections 2024-04-25T14:02:36.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7effbb430640 1 -- 172.21.15.150:0/3205056244 wait complete. 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 -- 172.21.15.150:0/2278533173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f84073f50 msgr2=0x7f3f840cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f84073f50 0x7f3f840cd860 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3f740097a0 tx=0x7f3f7402a180 comp rx=0 tx=0).stop 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 -- 172.21.15.150:0/2278533173 shutdown_connections 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f84073f50 0x7f3f840cd860 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 --2- 172.21.15.150:0/2278533173 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f840cfa80 0x7f3f840c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 -- 172.21.15.150:0/2278533173 >> 172.21.15.150:0/2278533173 conn(0x7f3f840b4130 msgr2=0x7f3f840d1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f27028de640 1 -- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc160860 msgr2=0x7f26fc160c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f27028de640 1 --2- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc160860 0x7f26fc160c70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0706a0 tx=0x7f26f402a6b0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 -- 172.21.15.150:0/924878458 shutdown_connections 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 --2- 172.21.15.150:0/924878458 >> v2:172.21.15.150:3300/0 conn(0x7f45d40b9600 0x7f45d415f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d406d8f0 0x7f45d40b90c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 --2- 172.21.15.150:0/924878458 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40d4380 0x7f45d40b4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 -- 172.21.15.150:0/924878458 >> 172.21.15.150:0/924878458 conn(0x7f45d40d2ca0 msgr2=0x7f45d40d5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 -- 172.21.15.150:0/803306267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4038c60 msgr2=0x7fd9a403afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 -- 172.21.15.150:0/2278533173 shutdown_connections 2024-04-25T14:02:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/803306267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4038c60 0x7fd9a403afb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40072d0 tx=0x7fd9a002a0d0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 -- 172.21.15.150:0/803306267 shutdown_connections 2024-04-25T14:02:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/803306267 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4038c60 0x7fd9a403afb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 -- 172.21.15.150:0/803306267 >> 172.21.15.150:0/803306267 conn(0x7fd9a4055e10 msgr2=0x7fd9a4056220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 -- 172.21.15.150:0/924878458 shutdown_connections 2024-04-25T14:02:36.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f073d516640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f073815f760 0x7f0738161b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f45d96b1640 1 -- 172.21.15.150:0/924878458 wait complete. 2024-04-25T14:02:36.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f073d516640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f073815f760 0x7f0738161b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38554/0 (socket says 172.21.15.150:38554) 2024-04-25T14:02:36.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f073d516640 1 -- 172.21.15.150:0/3895594356 learned_addr learned my addr 172.21.15.150:0/3895594356 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 -- 172.21.15.150:0/803306267 shutdown_connections 2024-04-25T14:02:36.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7fd9b1dff640 1 -- 172.21.15.150:0/803306267 wait complete. 2024-04-25T14:02:36.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f27028de640 1 -- 172.21.15.150:0/2047147340 shutdown_connections 2024-04-25T14:02:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0738162170 con 0x7f07380ce140 2024-04-25T14:02:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0738162450 con 0x7f073815e060 2024-04-25T14:02:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f3f89849640 1 -- 172.21.15.150:0/2278533173 wait complete. 2024-04-25T14:02:36.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0738162730 con 0x7f073815f760 2024-04-25T14:02:36.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.990+0000 7f27028de640 1 --2- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc160860 0x7f26fc160c70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f27028de640 1 --2- 172.21.15.150:0/2047147340 >> v2:172.21.15.150:3300/0 conn(0x7f26fc06d580 0x7f26fc06d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f27028de640 1 --2- 172.21.15.150:0/2047147340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc143120 0x7f26fc15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f27028de640 1 -- 172.21.15.150:0/2047147340 >> 172.21.15.150:0/2047147340 conn(0x7f26fc073090 msgr2=0x7f26fc0734a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f27028de640 1 -- 172.21.15.150:0/2047147340 shutdown_connections 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 0x7facc4151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facc41536d0 0x7facc4155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7facc40d1980 con 0x7facc415db90 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7facc40d1800 con 0x7facc41536d0 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7facca18f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facc40d1b00 con 0x7facc40cfdf0 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 0x7facc4151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 0x7facc4151a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38558/0 (socket says 172.21.15.150:38558) 2024-04-25T14:02:36.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.991+0000 7f27028de640 1 -- 172.21.15.150:0/2047147340 wait complete. 2024-04-25T14:02:36.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42242/0 (socket says 172.21.15.150:42242) 2024-04-25T14:02:36.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc2ffd640 1 -- 172.21.15.150:0/333480296 learned_addr learned my addr 172.21.15.150:0/333480296 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc3fff640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facc41536d0 0x7facc4155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1520966911 0 0) 0x7f0738162170 con 0x7f07380ce140 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0724007930 con 0x7f07380ce140 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 954098694 0 0) 0x7f0724007930 con 0x7f07380ce140 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 >> v2:172.21.15.150:3300/0 conn(0x7f073815f760 msgr2=0x7f0738161b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 --2- 172.21.15.150:0/3895594356 >> v2:172.21.15.150:3300/0 conn(0x7f073815f760 0x7f0738161b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 >> v1:172.21.15.187:6789/0 conn(0x7f073815e060 legacy=0x7f07380cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0738162a10 con 0x7f07380ce140 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f07300019f0 con 0x7f07380ce140 2024-04-25T14:02:36.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 >> v1:172.21.15.150:6789/0 conn(0x7f07380ce140 legacy=0x7f07380b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 -- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facc41536d0 msgr2=0x7facc4155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facc41536d0 0x7facc4155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 -- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 msgr2=0x7facc4151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 0x7facc4151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7facc37fe640 1 -- 172.21.15.150:0/333480296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facc414d0f0 con 0x7facc40cfdf0 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 0x7f0724008270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7f072ffff640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724008c10 0x7f0724009020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f072ffff640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0724009680 0x7f072400fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0738162730 con 0x7f0724009680 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07240105f0 con 0x7f0724007de0 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07240107f0 con 0x7f0724008c10 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 0x7f0724008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073dd17640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0724009680 0x7f072400fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7fd9b1dff640 1 Processor -- start 2024-04-25T14:02:36.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f3f89849640 1 Processor -- start 2024-04-25T14:02:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f45d96b1640 1 Processor -- start 2024-04-25T14:02:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 -- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724008c10 msgr2=0x7f0724009020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T14:02:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724008c10 0x7f0724009020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 -- 172.21.15.150:0/3895594356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0724009680 msgr2=0x7f072400fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f3f89849640 1 -- start start 2024-04-25T14:02:36.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.994+0000 7f45d96b1640 1 -- start start 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0724009680 0x7f072400fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073d516640 1 -- 172.21.15.150:0/3895594356 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0738162a10 con 0x7f0724007de0 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7f073dd17640 1 -- 172.21.15.150:0/3895594356 reap_dead start 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.994+0000 7f073d516640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 0x7f0724008270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0734009be0 tx=0x7f0724009c40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7e7d9c3ed779c33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.992+0000 7effbb430640 1 Processor -- start 2024-04-25T14:02:36.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7effbb430640 1 -- start start 2024-04-25T14:02:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.994+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0734036070 con 0x7f0724007de0 2024-04-25T14:02:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0734031400 con 0x7f0724007de0 2024-04-25T14:02:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7f072ffff640 1 -- 172.21.15.150:0/3895594356 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07340097e0 con 0x7f0724007de0 2024-04-25T14:02:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7facc37fe640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7facb400c410 tx=0x7facb400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.993+0000 7fd9b1dff640 1 -- start start 2024-04-25T14:02:36.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effbb430640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 0x7effb42060a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effbb430640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7f27028de640 1 Processor -- start 2024-04-25T14:02:36.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effbb430640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7f27028de640 1 -- start start 2024-04-25T14:02:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effb99cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effb99cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38578/0 (socket says 172.21.15.150:38578) 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.995+0000 7effb89cb640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effb89cb640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42250/0 (socket says 172.21.15.150:42250) 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effb99cd640 1 -- 172.21.15.150:0/1361519808 learned_addr learned my addr 172.21.15.150:0/1361519808 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b3860 con 0x7effb4203210 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 msgr2=0x7f0724008270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f073f77a640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 0x7f0724008270 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0734009be0 tx=0x7f0724009c40 comp rx=0 tx=0).stop 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 shutdown_connections 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f073f77a640 1 --2- 172.21.15.150:0/3895594356 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0724007de0 0x7f0724008270 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 >> 172.21.15.150:0/3895594356 conn(0x7f073809b4a0 msgr2=0x7f07380b6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb40074e0 con 0x7facc40cfdf0 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb40047e0 con 0x7facc40cfdf0 2024-04-25T14:02:36.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f45d96b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7effb40b36e0 con 0x7effb42042f0 2024-04-25T14:02:36.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effb91cc640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 0x7effb42060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f45d96b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40b9600 0x7f45d40729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f82ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f82ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38584/0 (socket says 172.21.15.150:38584) 2024-04-25T14:02:36.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f84164a80 0x7f3f840cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effb40b39e0 con 0x7effb4157d20 2024-04-25T14:02:36.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f82ffd640 1 -- 172.21.15.150:0/1014584924 learned_addr learned my addr 172.21.15.150:0/1014584924 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 0x7f3f8406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f840c7430 con 0x7f3f840cfa80 2024-04-25T14:02:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3f840c72b0 con 0x7f3f840b4bf0 2024-04-25T14:02:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3f841656d0 con 0x7f3f84164a80 2024-04-25T14:02:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f45d2ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38586/0 (socket says 172.21.15.150:38586) 2024-04-25T14:02:36.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.996+0000 7f3f837fe640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 0x7f3f8406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d2ffd640 1 -- 172.21.15.150:0/4110832767 learned_addr learned my addr 172.21.15.150:0/4110832767 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d40d4380 0x7f45d40ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d27fc640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40b9600 0x7f45d40729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 -- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 msgr2=0x7effb4158130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 -- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 msgr2=0x7effb42060a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 0x7effb42060a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb4158670 con 0x7effb4203210 2024-04-25T14:02:36.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f45d406dd00 con 0x7f45d406d8f0 2024-04-25T14:02:36.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f45d406d3d0 con 0x7f45d40d4380 2024-04-25T14:02:36.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f45d406de80 con 0x7f45d40b9600 2024-04-25T14:02:36.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f45d37fe640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d40d4380 0x7f45d40ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 shutdown_connections 2024-04-25T14:02:36.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f82ffd640 1 -- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f84164a80 msgr2=0x7f3f840cbab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T14:02:36.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facc414dab0 con 0x7facc40cfdf0 2024-04-25T14:02:36.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb4004bb0 con 0x7facc40cfdf0 2024-04-25T14:02:36.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb99cd640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7effb40b4230 tx=0x7effb000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f073f77a640 1 -- 172.21.15.150:0/3895594356 wait complete. 2024-04-25T14:02:36.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facc4152910 con 0x7facc40cfdf0 2024-04-25T14:02:36.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb91cc640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 0x7effb42060a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:36.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9a4038c60 0x7fd9a40ddf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7effb89cb640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:36.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 0x7fd9a40de4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 0x7fd9a40e2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a4007fd0 con 0x7fd9a4038c60 2024-04-25T14:02:36.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a4039070 con 0x7fd9a4186db0 2024-04-25T14:02:36.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7fd9b1dff640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a4008150 con 0x7fd9a40e1790 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f82ffd640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f84164a80 0x7f3f840cbab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f82ffd640 1 -- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 msgr2=0x7f3f8406d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f82ffd640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 0x7f3f8406d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f82ffd640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f840caa10 con 0x7f3f840cfa80 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.997+0000 7f3f837fe640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 0x7f3f8406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 0x7fd9a40e2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7fd9abfff640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 0x7fd9a40e2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:43454/0 (socket says 172.21.15.150:43454) 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7fd9abfff640 1 -- 172.21.15.150:0/352600621 learned_addr learned my addr 172.21.15.150:0/352600621 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7fd9ab7fe640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9a4038c60 0x7fd9a40ddf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 0x7fd9a40de4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f3f82ffd640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00ec10 tx=0x7f3f7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c018030 con 0x7f3f840cfa80 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f7c00cba0 con 0x7f3f840cfa80 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f7c003990 con 0x7f3f840cfa80 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb0017070 con 0x7effb4203210 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb000edf0 con 0x7effb4203210 2024-04-25T14:02:36.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb000c930 con 0x7effb4203210 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f27028de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 0x7f26fc141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f27028de640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc143120 0x7f26fc141f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f27028de640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f27028de640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0cdc80 con 0x7f26fc06d580 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f27028de640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f26fc0cd9a0 con 0x7f26fc143120 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f27028de640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26fc0cdf60 con 0x7f26fc160860 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f45d2ffd640 1 -- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40b9600 msgr2=0x7f45d40729f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f45d2ffd640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40b9600 0x7f45d40729f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f45d2ffd640 1 -- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d40d4380 msgr2=0x7f45d40ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f45d2ffd640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d40d4380 0x7f45d40ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.998+0000 7f45d2ffd640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d40710a0 con 0x7f45d406d8f0 2024-04-25T14:02:36.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facc4076a40 con 0x7facc40cfdf0 2024-04-25T14:02:36.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f45d2ffd640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e8a0 tx=0x7f45c400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb4203f60 con 0x7effb4203210 2024-04-25T14:02:36.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f840cacf0 con 0x7f3f840cfa80 2024-04-25T14:02:36.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f840cb2b0 con 0x7f3f840cfa80 2024-04-25T14:02:36.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb406d780 con 0x7effb4203210 2024-04-25T14:02:36.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c4017070 con 0x7f45d406d8f0 2024-04-25T14:02:36.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c400c940 con 0x7f45d406d8f0 2024-04-25T14:02:36.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c400cd30 con 0x7f45d406d8f0 2024-04-25T14:02:36.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 -- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 msgr2=0x7fd9a40e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7facb4011070 con 0x7facc40cfdf0 2024-04-25T14:02:36.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 0x7fd9a40e2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 -- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9a4038c60 msgr2=0x7fd9a40ddf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9a4038c60 0x7fd9a40ddf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7fd9aaffd640 1 -- 172.21.15.150:0/352600621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a404c480 con 0x7fd9a40e1790 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:35.999+0000 7f27020dd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:42304/0 (socket says 172.21.15.150:42304) 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 -- 172.21.15.150:0/1858329293 learned_addr learned my addr 172.21.15.150:0/1858329293 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27018dc640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 0x7f26fc141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27010db640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc143120 0x7f26fc141f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7fd9aaffd640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 0x7fd9a40de4d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d0b0 tx=0x7fd9ac069ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f073f77a640 1 Processor -- start 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 -- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc143120 msgr2=0x7f26fc141f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc143120 0x7f26fc141f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 -- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 msgr2=0x7f26fc141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 0x7f26fc141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 -- 172.21.15.150:0/1858329293 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26fc0b5880 con 0x7f26fc160860 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f073f77a640 1 -- start start 2024-04-25T14:02:36.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27018dc640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 0x7f26fc141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:36.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7facabfff640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 0x7facb0076a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.001+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f84050740 con 0x7f3f840cfa80 2024-04-25T14:02:36.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f27020dd640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26f4034040 tx=0x7f26f4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.001+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7facb4095010 con 0x7facc40cfdf0 2024-04-25T14:02:36.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.001+0000 7facc2ffd640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 0x7facb0076a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d4071380 con 0x7f45d406d8f0 2024-04-25T14:02:36.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.000+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d4071940 con 0x7f45d406d8f0 2024-04-25T14:02:36.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7facc2ffd640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 0x7facb0076a90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facc41eb8d0 tx=0x7facb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3f7c01b070 con 0x7f3f840cfa80 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4076a40 con 0x7effb4203210 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d4076a40 con 0x7f45d406d8f0 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724007de0 0x7f073814a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f073806d720 0x7f073806db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0738161f20 con 0x7f07380cfb30 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0738161da0 con 0x7f073806d720 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073f77a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f073815e470 con 0x7f0724007de0 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073d516640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724007de0 0x7f073814a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38618/0 (socket says 172.21.15.150:38618) 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 -- 172.21.15.150:0/943323232 learned_addr learned my addr 172.21.15.150:0/943323232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073dd17640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f073806d720 0x7f073806db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 -- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724007de0 msgr2=0x7f073814a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724007de0 0x7f073814a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 -- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f073806d720 msgr2=0x7f073806db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f073806d720 0x7f073806db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.002+0000 7f073cd15640 1 -- 172.21.15.150:0/943323232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07381474c0 con 0x7f07380cfb30 2024-04-25T14:02:36.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.003+0000 7f3f6bfff640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 0x7f3f70076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.003+0000 7f073cd15640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f072800c430 tx=0x7f072800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.003+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f3f7c095f40 con 0x7f3f840cfa80 2024-04-25T14:02:36.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.003+0000 7f3f827fc640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 0x7f3f70076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.003+0000 7f3f827fc640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 0x7f3f70076a40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f840b1390 tx=0x7f3f78007a40 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7effb001a070 con 0x7effb4203210 2024-04-25T14:02:36.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f45c401a070 con 0x7f45d406d8f0 2024-04-25T14:02:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7effa9ffb640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 0x7eff98076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7effb0095670 con 0x7effb4203210 2024-04-25T14:02:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7f45bb7fe640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 0x7f45c0076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.004+0000 7effb91cc640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 0x7eff98076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f45c4095ac0 con 0x7f45d406d8f0 2024-04-25T14:02:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f45d27fc640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 0x7f45c0076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4041020 con 0x7f26fc160860 2024-04-25T14:02:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac08ce40 con 0x7fd9a40e1790 2024-04-25T14:02:36.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ac090030 con 0x7fd9a40e1790 2024-04-25T14:02:36.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facb405f440 con 0x7facc40cfdf0 2024-04-25T14:02:36.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ac06d910 con 0x7fd9a40e1790 2024-04-25T14:02:36.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a41862d0 con 0x7fd9a40e1790 2024-04-25T14:02:36.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7effb91cc640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 0x7eff98076a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7effac033000 tx=0x7effac007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a4186890 con 0x7fd9a40e1790 2024-04-25T14:02:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f45d27fc640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 0x7f45c0076a40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45cc03c000 tx=0x7f45cc037000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f40077a0 con 0x7f26fc160860 2024-04-25T14:02:36.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f4036040 con 0x7f26fc160860 2024-04-25T14:02:36.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.005+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26fc152c40 con 0x7f26fc160860 2024-04-25T14:02:36.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.006+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26fc0b5dd0 con 0x7f26fc160860 2024-04-25T14:02:36.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a40059c0 con 0x7fd9a40e1790 2024-04-25T14:02:36.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26fc050740 con 0x7f26fc160860 2024-04-25T14:02:36.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd9ac06db70 con 0x7fd9a40e1790 2024-04-25T14:02:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7fd997fff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 0x7fd990076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f26f4044020 con 0x7f26fc160860 2024-04-25T14:02:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.007+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fd9ac111420 con 0x7fd9a40e1790 2024-04-25T14:02:36.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.008+0000 7fd9ab7fe640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 0x7fd990076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.008+0000 7f26f27fc640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 0x7f26e0076a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.008+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f26f40b6040 con 0x7f26fc160860 2024-04-25T14:02:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.008+0000 7f27018dc640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 0x7f26e0076a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.008+0000 7fd9ab7fe640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 0x7fd990076a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd9980014c0 tx=0x7fd998009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.009+0000 7f27018dc640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 0x7f26e0076a10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.010+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7effb4158670 con 0x7effb4203210 2024-04-25T14:02:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.012+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9ac0db950 con 0x7fd9a40e1790 2024-04-25T14:02:36.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.017+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26f404b030 con 0x7f26fc160860 2024-04-25T14:02:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.020+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f7c010040 con 0x7f3f840cfa80 2024-04-25T14:02:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.029+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728017070 con 0x7f07380cfb30 2024-04-25T14:02:36.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.030+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45c405f9a0 con 0x7f45d406d8f0 2024-04-25T14:02:36.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.029+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07280052f0 con 0x7f07380cfb30 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.029+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0728003830 con 0x7f07380cfb30 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.029+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f073806e070 con 0x7f07380cfb30 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.030+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07380d05b0 con 0x7f07380cfb30 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.033+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f072801a070 con 0x7f07380cfb30 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.034+0000 7f072dffb640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 0x7f071c076a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.034+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f07280946d0 con 0x7f07380cfb30 2024-04-25T14:02:36.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.034+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07380d0040 con 0x7f07380cfb30 2024-04-25T14:02:36.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.035+0000 7f073d516640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 0x7f071c076a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.036+0000 7f073d516640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 0x7f071c076a40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0734033000 tx=0x7f0734008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.045+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f073806e070 con 0x7f07380cfb30 2024-04-25T14:02:36.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.360+0000 7f61717fa640 1 -- 172.21.15.150:0/2912096895 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f616c005580 con 0x7f6184053de0 2024-04-25T14:02:36.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.360+0000 7f61737fe640 1 -- 172.21.15.150:0/2912096895 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f618c0dc700 con 0x7f6184053de0 2024-04-25T14:02:36.359 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:36.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 msgr2=0x7f6178076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 0x7f6178076a90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f617c034000 tx=0x7f617c02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 msgr2=0x7f61840dea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f618c06af00 tx=0x7f618c069ba0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 shutdown_connections 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6178074580 0x7f6178076a90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61840e0350 0x7f61840defb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6184053de0 0x7f61840dea70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 --2- 172.21.15.150:0/2912096895 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6184033950 0x7f618404bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.364+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 >> 172.21.15.150:0/2912096895 conn(0x7f6184054cf0 msgr2=0x7f61840e0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.365+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 shutdown_connections 2024-04-25T14:02:36.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.365+0000 7f618bfff640 1 -- 172.21.15.150:0/2912096895 wait complete. 2024-04-25T14:02:36.381 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.0 2024-04-25T14:02:36.381 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.397+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7facc406db00 con 0x7facc40cfdf0 2024-04-25T14:02:36.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.397+0000 7facabfff640 1 -- 172.21.15.150:0/333480296 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7facb405ed40 con 0x7facc40cfdf0 2024-04-25T14:02:36.396 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 msgr2=0x7facb0076a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 0x7facb0076a90 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7facc41eb8d0 tx=0x7facb8009040 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 msgr2=0x7facc414d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7facb400c410 tx=0x7facb400c8d0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 shutdown_connections 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facb0074580 0x7facb0076a90 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facc41536d0 0x7facc4155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facc415db90 0x7facc4151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 --2- 172.21.15.150:0/333480296 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facc40cfdf0 0x7facc414d570 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 >> 172.21.15.150:0/333480296 conn(0x7facc409b4a0 msgr2=0x7facc40b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 shutdown_connections 2024-04-25T14:02:36.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.400+0000 7facca18f640 1 -- 172.21.15.150:0/333480296 wait complete. 2024-04-25T14:02:36.411 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.7 2024-04-25T14:02:36.411 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.413+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd9a404c210 con 0x7fd9a40e1790 2024-04-25T14:02:36.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.413+0000 7fd997fff640 1 -- 172.21.15.150:0/352600621 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd9ac069cd0 con 0x7fd9a40e1790 2024-04-25T14:02:36.412 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 msgr2=0x7fd990076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 0x7fd990076a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd9980014c0 tx=0x7fd998009040 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 msgr2=0x7fd9a40de4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 0x7fd9a40de4d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd9a400d0b0 tx=0x7fd9ac069ba0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 shutdown_connections 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd990074500 0x7fd990076a10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9a4186db0 0x7fd9a40e2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd9a40e1790 0x7fd9a40de4d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 --2- 172.21.15.150:0/352600621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9a4038c60 0x7fd9a40ddf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 >> 172.21.15.150:0/352600621 conn(0x7fd9a4055e10 msgr2=0x7fd9a4008dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 shutdown_connections 2024-04-25T14:02:36.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.416+0000 7fd9b1dff640 1 -- 172.21.15.150:0/352600621 wait complete. 2024-04-25T14:02:36.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.428+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7effb40d2ca0 con 0x7effb4203210 2024-04-25T14:02:36.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.429+0000 7effa9ffb640 1 -- 172.21.15.150:0/1361519808 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7effb005fcc0 con 0x7effb4203210 2024-04-25T14:02:36.428 INFO:teuthology.orchestra.run.smithi150.stdout:55834574860 2024-04-25T14:02:36.431 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.4 2024-04-25T14:02:36.432 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 msgr2=0x7eff98076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 0x7eff98076a10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7effac033000 tx=0x7effac007000 comp rx=0 tx=0).stop 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 msgr2=0x7effb4203620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7effb40b4230 tx=0x7effb000ec00 comp rx=0 tx=0).stop 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 shutdown_connections 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff98074500 0x7eff98076a10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effb4203210 0x7effb4203620 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effb4157d20 0x7effb4158130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.432+0000 7effbb430640 1 --2- 172.21.15.150:0/1361519808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effb42042f0 0x7effb42060a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.433+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 >> 172.21.15.150:0/1361519808 conn(0x7effb40b76d0 msgr2=0x7effb415bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.433+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 shutdown_connections 2024-04-25T14:02:36.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.433+0000 7effbb430640 1 -- 172.21.15.150:0/1361519808 wait complete. 2024-04-25T14:02:36.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.439+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f26fc152e40 con 0x7f26fc160860 2024-04-25T14:02:36.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.440+0000 7f26f27fc640 1 -- 172.21.15.150:0/1858329293 <== mon.2 v2:172.21.15.187:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f26f404b030 con 0x7f26fc160860 2024-04-25T14:02:36.439 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-04-25T14:02:36.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.442+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f45d40d3f30 con 0x7f45d406d8f0 2024-04-25T14:02:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 msgr2=0x7f26e0076a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 0x7f26e0076a10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f26f8006fd0 tx=0x7f26f8008040 comp rx=0 tx=0).stop 2024-04-25T14:02:36.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f45bb7fe640 1 -- 172.21.15.150:0/4110832767 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f45c4063ca0 con 0x7f45d406d8f0 2024-04-25T14:02:36.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 msgr2=0x7f26fc153870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f26f4034040 tx=0x7f26f4004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:36.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 shutdown_connections 2024-04-25T14:02:36.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26e0074500 0x7f26e0076a10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.443 INFO:teuthology.orchestra.run.smithi150.stdout:55834574861 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f26fc160860 0x7f26fc153870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26fc143120 0x7f26fc141f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 --2- 172.21.15.150:0/1858329293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f26fc06d580 0x7f26fc141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 >> 172.21.15.150:0/1858329293 conn(0x7f26fc073090 msgr2=0x7f26fc06c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 shutdown_connections 2024-04-25T14:02:36.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.443+0000 7f27028de640 1 -- 172.21.15.150:0/1858329293 wait complete. 2024-04-25T14:02:36.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574860 got 55834574860 for osd.6 2024-04-25T14:02:36.445 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 msgr2=0x7f45c0076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 0x7f45c0076a40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45cc03c000 tx=0x7f45cc037000 comp rx=0 tx=0).stop 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 msgr2=0x7f45d40b42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f45c400e8a0 tx=0x7f45c400ed60 comp rx=0 tx=0).stop 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 shutdown_connections 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f45c0074530 0x7f45c0076a40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f45d40d4380 0x7f45d40ce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f45d40b9600 0x7f45d40729f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.446 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 --2- 172.21.15.150:0/4110832767 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f45d406d8f0 0x7f45d40b42a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 >> 172.21.15.150:0/4110832767 conn(0x7f45d40d2ca0 msgr2=0x7f45d40bab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 shutdown_connections 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.446+0000 7f45d96b1640 1 -- 172.21.15.150:0/4110832767 wait complete. 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.447+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f3f84074b50 con 0x7f3f840cfa80 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.448+0000 7f3f6bfff640 1 -- 172.21.15.150:0/1014584924 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3f7c010040 con 0x7f3f840cfa80 2024-04-25T14:02:36.447 INFO:teuthology.orchestra.run.smithi150.stdout:55834574859 2024-04-25T14:02:36.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 msgr2=0x7f3f70076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 0x7f3f70076a40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f840b1390 tx=0x7f3f78007a40 comp rx=0 tx=0).stop 2024-04-25T14:02:36.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 msgr2=0x7f3f84144ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c00ec10 tx=0x7f3f7c00c490 comp rx=0 tx=0).stop 2024-04-25T14:02:36.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 shutdown_connections 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f70074530 0x7f3f70076a40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f840b4bf0 0x7f3f8406d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f84164a80 0x7f3f840cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 --2- 172.21.15.150:0/1014584924 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f840cfa80 0x7f3f84144ec0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 >> 172.21.15.150:0/1014584924 conn(0x7f3f840b4130 msgr2=0x7f3f840b1c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 shutdown_connections 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.450+0000 7f3f89849640 1 -- 172.21.15.150:0/1014584924 wait complete. 2024-04-25T14:02:36.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.452+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f073815ee10 con 0x7f07380cfb30 2024-04-25T14:02:36.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.452+0000 7f072dffb640 1 -- 172.21.15.150:0/943323232 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f073815ee10 con 0x7f07380cfb30 2024-04-25T14:02:36.451 INFO:teuthology.orchestra.run.smithi150.stdout:55834574858 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.454+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 msgr2=0x7f071c076a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 0x7f071c076a40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0734033000 tx=0x7f0734008030 comp rx=0 tx=0).stop 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 msgr2=0x7f0738146f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f072800c430 tx=0x7f072800c830 comp rx=0 tx=0).stop 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 shutdown_connections 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f071c074530 0x7f071c076a40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f073806d720 0x7f073806db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07380cfb30 0x7f0738146f80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 --2- 172.21.15.150:0/943323232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0724007de0 0x7f073814a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 >> 172.21.15.150:0/943323232 conn(0x7f073809b4a0 msgr2=0x7f0738161170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 shutdown_connections 2024-04-25T14:02:36.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.455+0000 7f073f77a640 1 -- 172.21.15.150:0/943323232 wait complete. 2024-04-25T14:02:36.458 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.3 2024-04-25T14:02:36.458 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.460 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574861 got 55834574861 for osd.5 2024-04-25T14:02:36.460 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.464 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574859 for osd.2 2024-04-25T14:02:36.464 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.467 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-25T14:02:36.467 DEBUG:teuthology.parallel:result is None 2024-04-25T14:02:36.467 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T14:02:36.467 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:02:36.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.616+0000 7f9f0c9ba640 1 Processor -- start 2024-04-25T14:02:36.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.616+0000 7f9f0c9ba640 1 -- start start 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0c9ba640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9f080cfcc0 0x7f9f080b5400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0c9ba640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9f080b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0c9ba640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f080b5d50 con 0x7f9f081647a0 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0c9ba640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9f080b8380 con 0x7f9f080cfcc0 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0c9ba640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f08164bb0 con 0x7f9f080b5940 2024-04-25T14:02:36.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9f081647a0 0x7f9f08160070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55648/0 (socket says 172.21.15.150:55648) 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0659c640 1 -- 172.21.15.150:0/1741502819 learned_addr learned my addr 172.21.15.150:0/1741502819 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f06d9d640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9f080b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/1741502819 >> v2:172.21.15.187:3300/0 conn(0x7f9f080cfcc0 0x7f9f080b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928270159 0 0) 0x7f9ef8001490 con 0x7f9f081647a0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ef0003520 con 0x7f9f081647a0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2840224565 0 0) 0x7f9ef0003520 con 0x7f9f081647a0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 msgr2=0x7f9f080b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9f080b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 >> v2:172.21.15.187:3300/0 conn(0x7f9f080cfcc0 msgr2=0x7f9f080b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 --2- 172.21.15.150:0/1741502819 >> v2:172.21.15.187:3300/0 conn(0x7f9f080cfcc0 0x7f9f080b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.618+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f080d2c70 con 0x7f9f081647a0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f9ef8001bc0 con 0x7f9f081647a0 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 >> v1:172.21.15.150:6789/0 conn(0x7f9f081647a0 legacy=0x7f9f08160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef00039b0 0x7f9ef0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ef0004800 0x7f9ef0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9ef0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9f080b8380 con 0x7f9ef00039b0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f08164bb0 con 0x7f9ef0004800 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ef000ba70 con 0x7f9f080b5940 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef00039b0 0x7f9ef0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ef0004800 0x7f9ef0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f06d9d640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9ef0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 -- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 msgr2=0x7f9ef0005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9f080b5940 0x7f9ef0005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 -- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ef0004800 msgr2=0x7f9ef0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ef0004800 0x7f9ef0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.619+0000 7f9f0659c640 1 -- 172.21.15.150:0/1741502819 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef0003520 con 0x7f9ef00039b0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0659c640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef00039b0 0x7f9ef0003e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800aa40 tx=0x7f9ef0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2d673c94f5863f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef800a690 con 0x7f9ef00039b0 2024-04-25T14:02:36.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ef802f070 con 0x7f9ef00039b0 2024-04-25T14:02:36.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0559a640 1 -- 172.21.15.150:0/1741502819 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8034470 con 0x7f9ef00039b0 2024-04-25T14:02:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/1741502819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef00039b0 msgr2=0x7f9ef0003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.620+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/1741502819 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ef00039b0 0x7f9ef0003e20 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800aa40 tx=0x7f9ef0005830 comp rx=0 tx=0).stop 2024-04-25T14:02:36.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.621+0000 7f9f06d9d640 1 -- 172.21.15.150:0/1741502819 reap_dead start 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.621+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/1741502819 shutdown_connections 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.621+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/1741502819 >> 172.21.15.150:0/1741502819 conn(0x7f9f080d0400 msgr2=0x7f9f080d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.621+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/1741502819 shutdown_connections 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.621+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/1741502819 wait complete. 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.623+0000 7f9f0c9ba640 1 Processor -- start 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.623+0000 7f9f0c9ba640 1 -- start start 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ef00039b0 0x7f9f081f9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f080702e0 0x7f9f0806e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9f080d00d0 con 0x7f9f0806fed0 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9f080d3230 con 0x7f9f080702e0 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0c9ba640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9f080b7fa0 con 0x7f9ef00039b0 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38638/0 (socket says 172.21.15.150:38638) 2024-04-25T14:02:36.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 -- 172.21.15.150:0/831586571 learned_addr learned my addr 172.21.15.150:0/831586571 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f0659c640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ef00039b0 0x7f9f081f9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f06d9d640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f080702e0 0x7f9f0806e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 -- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ef00039b0 msgr2=0x7f9f081f9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ef00039b0 0x7f9f081f9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 -- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f080702e0 msgr2=0x7f9f0806e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f080702e0 0x7f9f0806e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.624+0000 7f9f05d9b640 1 -- 172.21.15.150:0/831586571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f081f6bf0 con 0x7f9f0806fed0 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9f05d9b640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9efc011930 tx=0x7f9efc011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc019070 con 0x7f9f0806fed0 2024-04-25T14:02:36.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9efc004240 con 0x7f9f0806fed0 2024-04-25T14:02:36.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9efc00f880 con 0x7f9f0806fed0 2024-04-25T14:02:36.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f08070ab0 con 0x7f9f0806fed0 2024-04-25T14:02:36.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.625+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0815ca60 con 0x7f9f0806fed0 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.626+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f08070820 con 0x7f9f0806fed0 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.627+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9efc01c070 con 0x7f9f0806fed0 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.627+0000 7f9ef6ffd640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 0x7f9ee80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.627+0000 7f9f0659c640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 0x7f9ee80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.627+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9efc0959e0 con 0x7f9f0806fed0 2024-04-25T14:02:36.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.628+0000 7f9f0659c640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 0x7f9ee80769c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800aa40 tx=0x7f9ef802d000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.632+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9efc05fdb0 con 0x7f9f0806fed0 2024-04-25T14:02:36.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.801+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f08076a40 con 0x7f9ee80744b0 2024-04-25T14:02:36.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.809+0000 7f9ef6ffd640 1 -- 172.21.15.150:0/831586571 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7f9ef8096300 con 0x7f9ee80744b0 2024-04-25T14:02:36.812 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:36.814 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 msgr2=0x7f9ee80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 0x7f9ee80769c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ef800aa40 tx=0x7f9ef802d000 comp rx=0 tx=0).stop 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 msgr2=0x7f9f0806df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9efc011930 tx=0x7f9efc011df0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 shutdown_connections 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ee80744b0 0x7f9ee80769c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9f080702e0 0x7f9f0806e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9f0806fed0 0x7f9f0806df10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 --2- 172.21.15.150:0/831586571 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ef00039b0 0x7f9f081f9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 >> 172.21.15.150:0/831586571 conn(0x7f9f080d0400 msgr2=0x7f9f081607b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 shutdown_connections 2024-04-25T14:02:36.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.813+0000 7f9f0c9ba640 1 -- 172.21.15.150:0/831586571 wait complete. 2024-04-25T14:02:36.827 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T14:02:36.592863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003244"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180578+0000","last_change":"2024-04-25T14:02:08.147737+0000","last_active":"2024-04-25T14:02:14.180578+0000","last_peered":"2024-04-25T14:02:14.180578+0000","last_clean":"2024-04-25T14:02:14.180578+0000","last_became_active":"2024-04-25T14:02:08.147072+0000","last_became_peered":"2024-04-25T14:02:08.147072+0000","last_unstale":"2024-04-25T14:02:14.180578+0000","last_undegraded":"2024-04-25T14:02:14.180578+0000","last_fullsized":"2024-04-25T14:02:14.180578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621342+0000","last_change":"2024-04-25T14:02:09.168014+0000","last_active":"2024-04-25T14:02:14.621342+0000","last_peered":"2024-04-25T14:02:14.621342+0000","last_clean":"2024-04-25T14:02:14.621342+0000","last_became_active":"2024-04-25T14:02:09.166846+0000","last_became_peered":"2024-04-25T14:02:09.166846+0000","last_unstale":"2024-04-25T14:02:14.621342+0000","last_undegraded":"2024-04-25T14:02:14.621342+0000","last_fullsized":"2024-04-25T14:02:14.621342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619101+0000","last_change":"2024-04-25T14:02:13.203189+0000","last_active":"2024-04-25T14:02:14.619101+0000","last_peered":"2024-04-25T14:02:14.619101+0000","last_clean":"2024-04-25T14:02:14.619101+0000","last_became_active":"2024-04-25T14:02:13.201743+0000","last_became_peered":"2024-04-25T14:02:13.201743+0000","last_unstale":"2024-04-25T14:02:14.619101+0000","last_undegraded":"2024-04-25T14:02:14.619101+0000","last_fullsized":"2024-04-25T14:02:14.619101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621459+0000","last_change":"2024-04-25T14:02:08.147961+0000","last_active":"2024-04-25T14:02:14.621459+0000","last_peered":"2024-04-25T14:02:14.621459+0000","last_clean":"2024-04-25T14:02:14.621459+0000","last_became_active":"2024-04-25T14:02:08.147652+0000","last_became_peered":"2024-04-25T14:02:08.147652+0000","last_unstale":"2024-04-25T14:02:14.621459+0000","last_undegraded":"2024-04-25T14:02:14.621459+0000","last_fullsized":"2024-04-25T14:02:14.621459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180307+0000","last_change":"2024-04-25T14:02:09.173887+0000","last_active":"2024-04-25T14:02:14.180307+0000","last_peered":"2024-04-25T14:02:14.180307+0000","last_clean":"2024-04-25T14:02:14.180307+0000","last_became_active":"2024-04-25T14:02:09.173583+0000","last_became_peered":"2024-04-25T14:02:09.173583+0000","last_unstale":"2024-04-25T14:02:14.180307+0000","last_undegraded":"2024-04-25T14:02:14.180307+0000","last_fullsized":"2024-04-25T14:02:14.180307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619293+0000","last_change":"2024-04-25T14:02:13.202095+0000","last_active":"2024-04-25T14:02:14.619293+0000","last_peered":"2024-04-25T14:02:14.619293+0000","last_clean":"2024-04-25T14:02:14.619293+0000","last_became_active":"2024-04-25T14:02:13.201620+0000","last_became_peered":"2024-04-25T14:02:13.201620+0000","last_unstale":"2024-04-25T14:02:14.619293+0000","last_undegraded":"2024-04-25T14:02:14.619293+0000","last_fullsized":"2024-04-25T14:02:14.619293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:25.777824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181439+0000","last_change":"2024-04-25T14:02:08.139279+0000","last_active":"2024-04-25T14:02:14.181439+0000","last_peered":"2024-04-25T14:02:14.181439+0000","last_clean":"2024-04-25T14:02:14.181439+0000","last_became_active":"2024-04-25T14:02:08.138923+0000","last_became_peered":"2024-04-25T14:02:08.138923+0000","last_unstale":"2024-04-25T14:02:14.181439+0000","last_undegraded":"2024-04-25T14:02:14.181439+0000","last_fullsized":"2024-04-25T14:02:14.181439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:22.987870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619690+0000","last_change":"2024-04-25T14:02:09.162551+0000","last_active":"2024-04-25T14:02:14.619690+0000","last_peered":"2024-04-25T14:02:14.619690+0000","last_clean":"2024-04-25T14:02:14.619690+0000","last_became_active":"2024-04-25T14:02:09.161794+0000","last_became_peered":"2024-04-25T14:02:09.161794+0000","last_unstale":"2024-04-25T14:02:14.619690+0000","last_undegraded":"2024-04-25T14:02:14.619690+0000","last_fullsized":"2024-04-25T14:02:14.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619746+0000","last_change":"2024-04-25T14:02:13.203879+0000","last_active":"2024-04-25T14:02:14.619746+0000","last_peered":"2024-04-25T14:02:14.619746+0000","last_clean":"2024-04-25T14:02:14.619746+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619746+0000","last_undegraded":"2024-04-25T14:02:14.619746+0000","last_fullsized":"2024-04-25T14:02:14.619746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.794247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180468+0000","last_change":"2024-04-25T14:02:08.148407+0000","last_active":"2024-04-25T14:02:14.180468+0000","last_peered":"2024-04-25T14:02:14.180468+0000","last_clean":"2024-04-25T14:02:14.180468+0000","last_became_active":"2024-04-25T14:02:08.148049+0000","last_became_peered":"2024-04-25T14:02:08.148049+0000","last_unstale":"2024-04-25T14:02:14.180468+0000","last_undegraded":"2024-04-25T14:02:14.180468+0000","last_fullsized":"2024-04-25T14:02:14.180468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620378+0000","last_change":"2024-04-25T14:02:09.162054+0000","last_active":"2024-04-25T14:02:14.620378+0000","last_peered":"2024-04-25T14:02:14.620378+0000","last_clean":"2024-04-25T14:02:14.620378+0000","last_became_active":"2024-04-25T14:02:09.161561+0000","last_became_peered":"2024-04-25T14:02:09.161561+0000","last_unstale":"2024-04-25T14:02:14.620378+0000","last_undegraded":"2024-04-25T14:02:14.620378+0000","last_fullsized":"2024-04-25T14:02:14.620378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620224+0000","last_change":"2024-04-25T14:02:13.202231+0000","last_active":"2024-04-25T14:02:14.620224+0000","last_peered":"2024-04-25T14:02:14.620224+0000","last_clean":"2024-04-25T14:02:14.620224+0000","last_became_active":"2024-04-25T14:02:13.201805+0000","last_became_peered":"2024-04-25T14:02:13.201805+0000","last_unstale":"2024-04-25T14:02:14.620224+0000","last_undegraded":"2024-04-25T14:02:14.620224+0000","last_fullsized":"2024-04-25T14:02:14.620224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:12.379500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181919+0000","last_change":"2024-04-25T14:02:09.164630+0000","last_active":"2024-04-25T14:02:14.181919+0000","last_peered":"2024-04-25T14:02:14.181919+0000","last_clean":"2024-04-25T14:02:14.181919+0000","last_became_active":"2024-04-25T14:02:09.164292+0000","last_became_peered":"2024-04-25T14:02:09.164292+0000","last_unstale":"2024-04-25T14:02:14.181919+0000","last_undegraded":"2024-04-25T14:02:14.181919+0000","last_fullsized":"2024-04-25T14:02:14.181919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180969+0000","last_change":"2024-04-25T14:02:08.136088+0000","last_active":"2024-04-25T14:02:14.180969+0000","last_peered":"2024-04-25T14:02:14.180969+0000","last_clean":"2024-04-25T14:02:14.180969+0000","last_became_active":"2024-04-25T14:02:08.135716+0000","last_became_peered":"2024-04-25T14:02:08.135716+0000","last_unstale":"2024-04-25T14:02:14.180969+0000","last_undegraded":"2024-04-25T14:02:14.180969+0000","last_fullsized":"2024-04-25T14:02:14.180969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:27.040903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620573+0000","last_change":"2024-04-25T14:02:13.196592+0000","last_active":"2024-04-25T14:02:14.620573+0000","last_peered":"2024-04-25T14:02:14.620573+0000","last_clean":"2024-04-25T14:02:14.620573+0000","last_became_active":"2024-04-25T14:02:13.196120+0000","last_became_peered":"2024-04-25T14:02:13.196120+0000","last_unstale":"2024-04-25T14:02:14.620573+0000","last_undegraded":"2024-04-25T14:02:14.620573+0000","last_fullsized":"2024-04-25T14:02:14.620573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620568+0000","last_change":"2024-04-25T14:02:09.163625+0000","last_active":"2024-04-25T14:02:14.620568+0000","last_peered":"2024-04-25T14:02:14.620568+0000","last_clean":"2024-04-25T14:02:14.620568+0000","last_became_active":"2024-04-25T14:02:09.163320+0000","last_became_peered":"2024-04-25T14:02:09.163320+0000","last_unstale":"2024-04-25T14:02:14.620568+0000","last_undegraded":"2024-04-25T14:02:14.620568+0000","last_fullsized":"2024-04-25T14:02:14.620568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619947+0000","last_change":"2024-04-25T14:02:08.140552+0000","last_active":"2024-04-25T14:02:14.619947+0000","last_peered":"2024-04-25T14:02:14.619947+0000","last_clean":"2024-04-25T14:02:14.619947+0000","last_became_active":"2024-04-25T14:02:08.139997+0000","last_became_peered":"2024-04-25T14:02:08.139997+0000","last_unstale":"2024-04-25T14:02:14.619947+0000","last_undegraded":"2024-04-25T14:02:14.619947+0000","last_fullsized":"2024-04-25T14:02:14.619947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620055+0000","last_change":"2024-04-25T14:02:13.204879+0000","last_active":"2024-04-25T14:02:14.620055+0000","last_peered":"2024-04-25T14:02:14.620055+0000","last_clean":"2024-04-25T14:02:14.620055+0000","last_became_active":"2024-04-25T14:02:13.203934+0000","last_became_peered":"2024-04-25T14:02:13.203934+0000","last_unstale":"2024-04-25T14:02:14.620055+0000","last_undegraded":"2024-04-25T14:02:14.620055+0000","last_fullsized":"2024-04-25T14:02:14.620055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619890+0000","last_change":"2024-04-25T14:02:09.157409+0000","last_active":"2024-04-25T14:02:14.619890+0000","last_peered":"2024-04-25T14:02:14.619890+0000","last_clean":"2024-04-25T14:02:14.619890+0000","last_became_active":"2024-04-25T14:02:09.157102+0000","last_became_peered":"2024-04-25T14:02:09.157102+0000","last_unstale":"2024-04-25T14:02:14.619890+0000","last_undegraded":"2024-04-25T14:02:14.619890+0000","last_fullsized":"2024-04-25T14:02:14.619890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619463+0000","last_change":"2024-04-25T14:02:08.138257+0000","last_active":"2024-04-25T14:02:14.619463+0000","last_peered":"2024-04-25T14:02:14.619463+0000","last_clean":"2024-04-25T14:02:14.619463+0000","last_became_active":"2024-04-25T14:02:08.137925+0000","last_became_peered":"2024-04-25T14:02:08.137925+0000","last_unstale":"2024-04-25T14:02:14.619463+0000","last_undegraded":"2024-04-25T14:02:14.619463+0000","last_fullsized":"2024-04-25T14:02:14.619463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619999+0000","last_change":"2024-04-25T14:02:13.204203+0000","last_active":"2024-04-25T14:02:14.619999+0000","last_peered":"2024-04-25T14:02:14.619999+0000","last_clean":"2024-04-25T14:02:14.619999+0000","last_became_active":"2024-04-25T14:02:13.203578+0000","last_became_peered":"2024-04-25T14:02:13.203578+0000","last_unstale":"2024-04-25T14:02:14.619999+0000","last_undegraded":"2024-04-25T14:02:14.619999+0000","last_fullsized":"2024-04-25T14:02:14.619999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620454+0000","last_change":"2024-04-25T14:02:09.153035+0000","last_active":"2024-04-25T14:02:14.620454+0000","last_peered":"2024-04-25T14:02:14.620454+0000","last_clean":"2024-04-25T14:02:14.620454+0000","last_became_active":"2024-04-25T14:02:09.152071+0000","last_became_peered":"2024-04-25T14:02:09.152071+0000","last_unstale":"2024-04-25T14:02:14.620454+0000","last_undegraded":"2024-04-25T14:02:14.620454+0000","last_fullsized":"2024-04-25T14:02:14.620454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:11.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180075+0000","last_change":"2024-04-25T14:02:08.141740+0000","last_active":"2024-04-25T14:02:14.180075+0000","last_peered":"2024-04-25T14:02:14.180075+0000","last_clean":"2024-04-25T14:02:14.180075+0000","last_became_active":"2024-04-25T14:02:08.141400+0000","last_became_peered":"2024-04-25T14:02:08.141400+0000","last_unstale":"2024-04-25T14:02:14.180075+0000","last_undegraded":"2024-04-25T14:02:14.180075+0000","last_fullsized":"2024-04-25T14:02:14.180075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619288+0000","last_change":"2024-04-25T14:02:13.202697+0000","last_active":"2024-04-25T14:02:14.619288+0000","last_peered":"2024-04-25T14:02:14.619288+0000","last_clean":"2024-04-25T14:02:14.619288+0000","last_became_active":"2024-04-25T14:02:13.201555+0000","last_became_peered":"2024-04-25T14:02:13.201555+0000","last_unstale":"2024-04-25T14:02:14.619288+0000","last_undegraded":"2024-04-25T14:02:14.619288+0000","last_fullsized":"2024-04-25T14:02:14.619288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:43.769087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180683+0000","last_change":"2024-04-25T14:02:09.166496+0000","last_active":"2024-04-25T14:02:14.180683+0000","last_peered":"2024-04-25T14:02:14.180683+0000","last_clean":"2024-04-25T14:02:14.180683+0000","last_became_active":"2024-04-25T14:02:09.166180+0000","last_became_peered":"2024-04-25T14:02:09.166180+0000","last_unstale":"2024-04-25T14:02:14.180683+0000","last_undegraded":"2024-04-25T14:02:14.180683+0000","last_fullsized":"2024-04-25T14:02:14.180683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181386+0000","last_change":"2024-04-25T14:02:08.143988+0000","last_active":"2024-04-25T14:02:14.181386+0000","last_peered":"2024-04-25T14:02:14.181386+0000","last_clean":"2024-04-25T14:02:14.181386+0000","last_became_active":"2024-04-25T14:02:08.143639+0000","last_became_peered":"2024-04-25T14:02:08.143639+0000","last_unstale":"2024-04-25T14:02:14.181386+0000","last_undegraded":"2024-04-25T14:02:14.181386+0000","last_fullsized":"2024-04-25T14:02:14.181386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620239+0000","last_change":"2024-04-25T14:02:13.193130+0000","last_active":"2024-04-25T14:02:14.620239+0000","last_peered":"2024-04-25T14:02:14.620239+0000","last_clean":"2024-04-25T14:02:14.620239+0000","last_became_active":"2024-04-25T14:02:13.192664+0000","last_became_peered":"2024-04-25T14:02:13.192664+0000","last_unstale":"2024-04-25T14:02:14.620239+0000","last_undegraded":"2024-04-25T14:02:14.620239+0000","last_fullsized":"2024-04-25T14:02:14.620239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621118+0000","last_change":"2024-04-25T14:02:09.175890+0000","last_active":"2024-04-25T14:02:14.621118+0000","last_peered":"2024-04-25T14:02:14.621118+0000","last_clean":"2024-04-25T14:02:14.621118+0000","last_became_active":"2024-04-25T14:02:09.175586+0000","last_became_peered":"2024-04-25T14:02:09.175586+0000","last_unstale":"2024-04-25T14:02:14.621118+0000","last_undegraded":"2024-04-25T14:02:14.621118+0000","last_fullsized":"2024-04-25T14:02:14.621118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619271+0000","last_change":"2024-04-25T14:02:08.141597+0000","last_active":"2024-04-25T14:02:14.619271+0000","last_peered":"2024-04-25T14:02:14.619271+0000","last_clean":"2024-04-25T14:02:14.619271+0000","last_became_active":"2024-04-25T14:02:08.141271+0000","last_became_peered":"2024-04-25T14:02:08.141271+0000","last_unstale":"2024-04-25T14:02:14.619271+0000","last_undegraded":"2024-04-25T14:02:14.619271+0000","last_fullsized":"2024-04-25T14:02:14.619271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619661+0000","last_change":"2024-04-25T14:02:13.208359+0000","last_active":"2024-04-25T14:02:14.619661+0000","last_peered":"2024-04-25T14:02:14.619661+0000","last_clean":"2024-04-25T14:02:14.619661+0000","last_became_active":"2024-04-25T14:02:13.204049+0000","last_became_peered":"2024-04-25T14:02:13.204049+0000","last_unstale":"2024-04-25T14:02:14.619661+0000","last_undegraded":"2024-04-25T14:02:14.619661+0000","last_fullsized":"2024-04-25T14:02:14.619661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620199+0000","last_change":"2024-04-25T14:02:09.153508+0000","last_active":"2024-04-25T14:02:14.620199+0000","last_peered":"2024-04-25T14:02:14.620199+0000","last_clean":"2024-04-25T14:02:14.620199+0000","last_became_active":"2024-04-25T14:02:09.153087+0000","last_became_peered":"2024-04-25T14:02:09.153087+0000","last_unstale":"2024-04-25T14:02:14.620199+0000","last_undegraded":"2024-04-25T14:02:14.620199+0000","last_fullsized":"2024-04-25T14:02:14.620199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181562+0000","last_change":"2024-04-25T14:02:08.149363+0000","last_active":"2024-04-25T14:02:14.181562+0000","last_peered":"2024-04-25T14:02:14.181562+0000","last_clean":"2024-04-25T14:02:14.181562+0000","last_became_active":"2024-04-25T14:02:08.148936+0000","last_became_peered":"2024-04-25T14:02:08.148936+0000","last_unstale":"2024-04-25T14:02:14.181562+0000","last_undegraded":"2024-04-25T14:02:14.181562+0000","last_fullsized":"2024-04-25T14:02:14.181562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:33.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180089+0000","last_change":"2024-04-25T14:02:13.201594+0000","last_active":"2024-04-25T14:02:14.180089+0000","last_peered":"2024-04-25T14:02:14.180089+0000","last_clean":"2024-04-25T14:02:14.180089+0000","last_became_active":"2024-04-25T14:02:13.201156+0000","last_became_peered":"2024-04-25T14:02:13.201156+0000","last_unstale":"2024-04-25T14:02:14.180089+0000","last_undegraded":"2024-04-25T14:02:14.180089+0000","last_fullsized":"2024-04-25T14:02:14.180089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620504+0000","last_change":"2024-04-25T14:02:09.164648+0000","last_active":"2024-04-25T14:02:14.620504+0000","last_peered":"2024-04-25T14:02:14.620504+0000","last_clean":"2024-04-25T14:02:14.620504+0000","last_became_active":"2024-04-25T14:02:09.164319+0000","last_became_peered":"2024-04-25T14:02:09.164319+0000","last_unstale":"2024-04-25T14:02:14.620504+0000","last_undegraded":"2024-04-25T14:02:14.620504+0000","last_fullsized":"2024-04-25T14:02:14.620504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620863+0000","last_change":"2024-04-25T14:02:08.148046+0000","last_active":"2024-04-25T14:02:14.620863+0000","last_peered":"2024-04-25T14:02:14.620863+0000","last_clean":"2024-04-25T14:02:14.620863+0000","last_became_active":"2024-04-25T14:02:08.147707+0000","last_became_peered":"2024-04-25T14:02:08.147707+0000","last_unstale":"2024-04-25T14:02:14.620863+0000","last_undegraded":"2024-04-25T14:02:14.620863+0000","last_fullsized":"2024-04-25T14:02:14.620863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619751+0000","last_change":"2024-04-25T14:02:13.197572+0000","last_active":"2024-04-25T14:02:14.619751+0000","last_peered":"2024-04-25T14:02:14.619751+0000","last_clean":"2024-04-25T14:02:14.619751+0000","last_became_active":"2024-04-25T14:02:13.197138+0000","last_became_peered":"2024-04-25T14:02:13.197138+0000","last_unstale":"2024-04-25T14:02:14.619751+0000","last_undegraded":"2024-04-25T14:02:14.619751+0000","last_fullsized":"2024-04-25T14:02:14.619751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619861+0000","last_change":"2024-04-25T14:02:09.163807+0000","last_active":"2024-04-25T14:02:14.619861+0000","last_peered":"2024-04-25T14:02:14.619861+0000","last_clean":"2024-04-25T14:02:14.619861+0000","last_became_active":"2024-04-25T14:02:09.163510+0000","last_became_peered":"2024-04-25T14:02:09.163510+0000","last_unstale":"2024-04-25T14:02:14.619861+0000","last_undegraded":"2024-04-25T14:02:14.619861+0000","last_fullsized":"2024-04-25T14:02:14.619861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620015+0000","last_change":"2024-04-25T14:02:08.148684+0000","last_active":"2024-04-25T14:02:14.620015+0000","last_peered":"2024-04-25T14:02:14.620015+0000","last_clean":"2024-04-25T14:02:14.620015+0000","last_became_active":"2024-04-25T14:02:08.148384+0000","last_became_peered":"2024-04-25T14:02:08.148384+0000","last_unstale":"2024-04-25T14:02:14.620015+0000","last_undegraded":"2024-04-25T14:02:14.620015+0000","last_fullsized":"2024-04-25T14:02:14.620015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620837+0000","last_change":"2024-04-25T14:02:13.204849+0000","last_active":"2024-04-25T14:02:14.620837+0000","last_peered":"2024-04-25T14:02:14.620837+0000","last_clean":"2024-04-25T14:02:14.620837+0000","last_became_active":"2024-04-25T14:02:13.204459+0000","last_became_peered":"2024-04-25T14:02:13.204459+0000","last_unstale":"2024-04-25T14:02:14.620837+0000","last_undegraded":"2024-04-25T14:02:14.620837+0000","last_fullsized":"2024-04-25T14:02:14.620837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619156+0000","last_change":"2024-04-25T14:02:09.153461+0000","last_active":"2024-04-25T14:02:14.619156+0000","last_peered":"2024-04-25T14:02:14.619156+0000","last_clean":"2024-04-25T14:02:14.619156+0000","last_became_active":"2024-04-25T14:02:09.153122+0000","last_became_peered":"2024-04-25T14:02:09.153122+0000","last_unstale":"2024-04-25T14:02:14.619156+0000","last_undegraded":"2024-04-25T14:02:14.619156+0000","last_fullsized":"2024-04-25T14:02:14.619156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619253+0000","last_change":"2024-04-25T14:02:08.141488+0000","last_active":"2024-04-25T14:02:14.619253+0000","last_peered":"2024-04-25T14:02:14.619253+0000","last_clean":"2024-04-25T14:02:14.619253+0000","last_became_active":"2024-04-25T14:02:08.141195+0000","last_became_peered":"2024-04-25T14:02:08.141195+0000","last_unstale":"2024-04-25T14:02:14.619253+0000","last_undegraded":"2024-04-25T14:02:14.619253+0000","last_fullsized":"2024-04-25T14:02:14.619253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620173+0000","last_change":"2024-04-25T14:02:13.202926+0000","last_active":"2024-04-25T14:02:14.620173+0000","last_peered":"2024-04-25T14:02:14.620173+0000","last_clean":"2024-04-25T14:02:14.620173+0000","last_became_active":"2024-04-25T14:02:13.202452+0000","last_became_peered":"2024-04-25T14:02:13.202452+0000","last_unstale":"2024-04-25T14:02:14.620173+0000","last_undegraded":"2024-04-25T14:02:14.620173+0000","last_fullsized":"2024-04-25T14:02:14.620173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179693+0000","last_change":"2024-04-25T14:02:09.162133+0000","last_active":"2024-04-25T14:02:14.179693+0000","last_peered":"2024-04-25T14:02:14.179693+0000","last_clean":"2024-04-25T14:02:14.179693+0000","last_became_active":"2024-04-25T14:02:09.161795+0000","last_became_peered":"2024-04-25T14:02:09.161795+0000","last_unstale":"2024-04-25T14:02:14.179693+0000","last_undegraded":"2024-04-25T14:02:14.179693+0000","last_fullsized":"2024-04-25T14:02:14.179693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:49.968260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180083+0000","last_change":"2024-04-25T14:02:08.144670+0000","last_active":"2024-04-25T14:02:14.180083+0000","last_peered":"2024-04-25T14:02:14.180083+0000","last_clean":"2024-04-25T14:02:14.180083+0000","last_became_active":"2024-04-25T14:02:08.143303+0000","last_became_peered":"2024-04-25T14:02:08.143303+0000","last_unstale":"2024-04-25T14:02:14.180083+0000","last_undegraded":"2024-04-25T14:02:14.180083+0000","last_fullsized":"2024-04-25T14:02:14.180083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.199069+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.198521+0000","last_became_peered":"2024-04-25T14:02:13.198521+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.164561+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.164215+0000","last_became_peered":"2024-04-25T14:02:09.164215+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619618+0000","last_change":"2024-04-25T14:02:08.150157+0000","last_active":"2024-04-25T14:02:14.619618+0000","last_peered":"2024-04-25T14:02:14.619618+0000","last_clean":"2024-04-25T14:02:14.619618+0000","last_became_active":"2024-04-25T14:02:08.149812+0000","last_became_peered":"2024-04-25T14:02:08.149812+0000","last_unstale":"2024-04-25T14:02:14.619618+0000","last_undegraded":"2024-04-25T14:02:14.619618+0000","last_fullsized":"2024-04-25T14:02:14.619618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180256+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:02:14.180256+0000","last_peered":"2024-04-25T14:02:14.180256+0000","last_clean":"2024-04-25T14:02:14.180256+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:02:14.180256+0000","last_undegraded":"2024-04-25T14:02:14.180256+0000","last_fullsized":"2024-04-25T14:02:14.180256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.202985+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.202514+0000","last_became_peered":"2024-04-25T14:02:13.202514+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181284+0000","last_change":"2024-04-25T14:02:09.164697+0000","last_active":"2024-04-25T14:02:14.181284+0000","last_peered":"2024-04-25T14:02:14.181284+0000","last_clean":"2024-04-25T14:02:14.181284+0000","last_became_active":"2024-04-25T14:02:09.164352+0000","last_became_peered":"2024-04-25T14:02:09.164352+0000","last_unstale":"2024-04-25T14:02:14.181284+0000","last_undegraded":"2024-04-25T14:02:14.181284+0000","last_fullsized":"2024-04-25T14:02:14.181284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620427+0000","last_change":"2024-04-25T14:02:08.150437+0000","last_active":"2024-04-25T14:02:14.620427+0000","last_peered":"2024-04-25T14:02:14.620427+0000","last_clean":"2024-04-25T14:02:14.620427+0000","last_became_active":"2024-04-25T14:02:08.149954+0000","last_became_peered":"2024-04-25T14:02:08.149954+0000","last_unstale":"2024-04-25T14:02:14.620427+0000","last_undegraded":"2024-04-25T14:02:14.620427+0000","last_fullsized":"2024-04-25T14:02:14.620427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181863+0000","last_change":"2024-04-25T14:02:13.198918+0000","last_active":"2024-04-25T14:02:14.181863+0000","last_peered":"2024-04-25T14:02:14.181863+0000","last_clean":"2024-04-25T14:02:14.181863+0000","last_became_active":"2024-04-25T14:02:13.198244+0000","last_became_peered":"2024-04-25T14:02:13.198244+0000","last_unstale":"2024-04-25T14:02:14.181863+0000","last_undegraded":"2024-04-25T14:02:14.181863+0000","last_fullsized":"2024-04-25T14:02:14.181863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619706+0000","last_change":"2024-04-25T14:02:09.162899+0000","last_active":"2024-04-25T14:02:14.619706+0000","last_peered":"2024-04-25T14:02:14.619706+0000","last_clean":"2024-04-25T14:02:14.619706+0000","last_became_active":"2024-04-25T14:02:09.162573+0000","last_became_peered":"2024-04-25T14:02:09.162573+0000","last_unstale":"2024-04-25T14:02:14.619706+0000","last_undegraded":"2024-04-25T14:02:14.619706+0000","last_fullsized":"2024-04-25T14:02:14.619706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621027+0000","last_change":"2024-04-25T14:02:08.140157+0000","last_active":"2024-04-25T14:02:14.621027+0000","last_peered":"2024-04-25T14:02:14.621027+0000","last_clean":"2024-04-25T14:02:14.621027+0000","last_became_active":"2024-04-25T14:02:08.139477+0000","last_became_peered":"2024-04-25T14:02:08.139477+0000","last_unstale":"2024-04-25T14:02:14.621027+0000","last_undegraded":"2024-04-25T14:02:14.621027+0000","last_fullsized":"2024-04-25T14:02:14.621027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180180+0000","last_change":"2024-04-25T14:02:13.201803+0000","last_active":"2024-04-25T14:02:14.180180+0000","last_peered":"2024-04-25T14:02:14.180180+0000","last_clean":"2024-04-25T14:02:14.180180+0000","last_became_active":"2024-04-25T14:02:13.201388+0000","last_became_peered":"2024-04-25T14:02:13.201388+0000","last_unstale":"2024-04-25T14:02:14.180180+0000","last_undegraded":"2024-04-25T14:02:14.180180+0000","last_fullsized":"2024-04-25T14:02:14.180180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182344+0000","last_change":"2024-04-25T14:02:09.171757+0000","last_active":"2024-04-25T14:02:14.182344+0000","last_peered":"2024-04-25T14:02:14.182344+0000","last_clean":"2024-04-25T14:02:14.182344+0000","last_became_active":"2024-04-25T14:02:09.168908+0000","last_became_peered":"2024-04-25T14:02:09.168908+0000","last_unstale":"2024-04-25T14:02:14.182344+0000","last_undegraded":"2024-04-25T14:02:14.182344+0000","last_fullsized":"2024-04-25T14:02:14.182344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:21.339622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620710+0000","last_change":"2024-04-25T14:02:08.140886+0000","last_active":"2024-04-25T14:02:14.620710+0000","last_peered":"2024-04-25T14:02:14.620710+0000","last_clean":"2024-04-25T14:02:14.620710+0000","last_became_active":"2024-04-25T14:02:08.138585+0000","last_became_peered":"2024-04-25T14:02:08.138585+0000","last_unstale":"2024-04-25T14:02:14.620710+0000","last_undegraded":"2024-04-25T14:02:14.620710+0000","last_fullsized":"2024-04-25T14:02:14.620710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619005+0000","last_change":"2024-04-25T14:02:13.194033+0000","last_active":"2024-04-25T14:02:14.619005+0000","last_peered":"2024-04-25T14:02:14.619005+0000","last_clean":"2024-04-25T14:02:14.619005+0000","last_became_active":"2024-04-25T14:02:13.193486+0000","last_became_peered":"2024-04-25T14:02:13.193486+0000","last_unstale":"2024-04-25T14:02:14.619005+0000","last_undegraded":"2024-04-25T14:02:14.619005+0000","last_fullsized":"2024-04-25T14:02:14.619005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181242+0000","last_change":"2024-04-25T14:02:09.154176+0000","last_active":"2024-04-25T14:02:14.181242+0000","last_peered":"2024-04-25T14:02:14.181242+0000","last_clean":"2024-04-25T14:02:14.181242+0000","last_became_active":"2024-04-25T14:02:09.153867+0000","last_became_peered":"2024-04-25T14:02:09.153867+0000","last_unstale":"2024-04-25T14:02:14.181242+0000","last_undegraded":"2024-04-25T14:02:14.181242+0000","last_fullsized":"2024-04-25T14:02:14.181242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619639+0000","last_change":"2024-04-25T14:02:08.148162+0000","last_active":"2024-04-25T14:02:14.619639+0000","last_peered":"2024-04-25T14:02:14.619639+0000","last_clean":"2024-04-25T14:02:14.619639+0000","last_became_active":"2024-04-25T14:02:08.147822+0000","last_became_peered":"2024-04-25T14:02:08.147822+0000","last_unstale":"2024-04-25T14:02:14.619639+0000","last_undegraded":"2024-04-25T14:02:14.619639+0000","last_fullsized":"2024-04-25T14:02:14.619639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:47.001110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619644+0000","last_change":"2024-04-25T14:02:13.204464+0000","last_active":"2024-04-25T14:02:14.619644+0000","last_peered":"2024-04-25T14:02:14.619644+0000","last_clean":"2024-04-25T14:02:14.619644+0000","last_became_active":"2024-04-25T14:02:13.203501+0000","last_became_peered":"2024-04-25T14:02:13.203501+0000","last_unstale":"2024-04-25T14:02:14.619644+0000","last_undegraded":"2024-04-25T14:02:14.619644+0000","last_fullsized":"2024-04-25T14:02:14.619644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:36.450612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180471+0000","last_change":"2024-04-25T14:02:09.169427+0000","last_active":"2024-04-25T14:02:14.180471+0000","last_peered":"2024-04-25T14:02:14.180471+0000","last_clean":"2024-04-25T14:02:14.180471+0000","last_became_active":"2024-04-25T14:02:09.168920+0000","last_became_peered":"2024-04-25T14:02:09.168920+0000","last_unstale":"2024-04-25T14:02:14.180471+0000","last_undegraded":"2024-04-25T14:02:14.180471+0000","last_fullsized":"2024-04-25T14:02:14.180471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618690+0000","last_change":"2024-04-25T14:02:08.143121+0000","last_active":"2024-04-25T14:02:14.618690+0000","last_peered":"2024-04-25T14:02:14.618690+0000","last_clean":"2024-04-25T14:02:14.618690+0000","last_became_active":"2024-04-25T14:02:08.142765+0000","last_became_peered":"2024-04-25T14:02:08.142765+0000","last_unstale":"2024-04-25T14:02:14.618690+0000","last_undegraded":"2024-04-25T14:02:14.618690+0000","last_fullsized":"2024-04-25T14:02:14.618690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180517+0000","last_change":"2024-04-25T14:02:13.205101+0000","last_active":"2024-04-25T14:02:14.180517+0000","last_peered":"2024-04-25T14:02:14.180517+0000","last_clean":"2024-04-25T14:02:14.180517+0000","last_became_active":"2024-04-25T14:02:13.204696+0000","last_became_peered":"2024-04-25T14:02:13.204696+0000","last_unstale":"2024-04-25T14:02:14.180517+0000","last_undegraded":"2024-04-25T14:02:14.180517+0000","last_fullsized":"2024-04-25T14:02:14.180517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179895+0000","last_change":"2024-04-25T14:02:09.165439+0000","last_active":"2024-04-25T14:02:14.179895+0000","last_peered":"2024-04-25T14:02:14.179895+0000","last_clean":"2024-04-25T14:02:14.179895+0000","last_became_active":"2024-04-25T14:02:09.164934+0000","last_became_peered":"2024-04-25T14:02:09.164934+0000","last_unstale":"2024-04-25T14:02:14.179895+0000","last_undegraded":"2024-04-25T14:02:14.179895+0000","last_fullsized":"2024-04-25T14:02:14.179895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180810+0000","last_change":"2024-04-25T14:02:08.144013+0000","last_active":"2024-04-25T14:02:14.180810+0000","last_peered":"2024-04-25T14:02:14.180810+0000","last_clean":"2024-04-25T14:02:14.180810+0000","last_became_active":"2024-04-25T14:02:08.141723+0000","last_became_peered":"2024-04-25T14:02:08.141723+0000","last_unstale":"2024-04-25T14:02:14.180810+0000","last_undegraded":"2024-04-25T14:02:14.180810+0000","last_fullsized":"2024-04-25T14:02:14.180810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620974+0000","last_change":"2024-04-25T14:02:13.199511+0000","last_active":"2024-04-25T14:02:14.620974+0000","last_peered":"2024-04-25T14:02:14.620974+0000","last_clean":"2024-04-25T14:02:14.620974+0000","last_became_active":"2024-04-25T14:02:13.199074+0000","last_became_peered":"2024-04-25T14:02:13.199074+0000","last_unstale":"2024-04-25T14:02:14.620974+0000","last_undegraded":"2024-04-25T14:02:14.620974+0000","last_fullsized":"2024-04-25T14:02:14.620974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:47.660769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180811+0000","last_change":"2024-04-25T14:02:09.171097+0000","last_active":"2024-04-25T14:02:14.180811+0000","last_peered":"2024-04-25T14:02:14.180811+0000","last_clean":"2024-04-25T14:02:14.180811+0000","last_became_active":"2024-04-25T14:02:09.169656+0000","last_became_peered":"2024-04-25T14:02:09.169656+0000","last_unstale":"2024-04-25T14:02:14.180811+0000","last_undegraded":"2024-04-25T14:02:14.180811+0000","last_fullsized":"2024-04-25T14:02:14.180811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:34.914414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619583+0000","last_change":"2024-04-25T14:02:08.148427+0000","last_active":"2024-04-25T14:02:14.619583+0000","last_peered":"2024-04-25T14:02:14.619583+0000","last_clean":"2024-04-25T14:02:14.619583+0000","last_became_active":"2024-04-25T14:02:08.148019+0000","last_became_peered":"2024-04-25T14:02:08.148019+0000","last_unstale":"2024-04-25T14:02:14.619583+0000","last_undegraded":"2024-04-25T14:02:14.619583+0000","last_fullsized":"2024-04-25T14:02:14.619583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621238+0000","last_change":"2024-04-25T14:02:13.203865+0000","last_active":"2024-04-25T14:02:14.621238+0000","last_peered":"2024-04-25T14:02:14.621238+0000","last_clean":"2024-04-25T14:02:14.621238+0000","last_became_active":"2024-04-25T14:02:13.203056+0000","last_became_peered":"2024-04-25T14:02:13.203056+0000","last_unstale":"2024-04-25T14:02:14.621238+0000","last_undegraded":"2024-04-25T14:02:14.621238+0000","last_fullsized":"2024-04-25T14:02:14.621238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:58.555213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179953+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.179953+0000","last_peered":"2024-04-25T14:02:14.179953+0000","last_clean":"2024-04-25T14:02:14.179953+0000","last_became_active":"2024-04-25T14:02:09.171025+0000","last_became_peered":"2024-04-25T14:02:09.171025+0000","last_unstale":"2024-04-25T14:02:14.179953+0000","last_undegraded":"2024-04-25T14:02:14.179953+0000","last_fullsized":"2024-04-25T14:02:14.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180403+0000","last_change":"2024-04-25T14:02:08.141709+0000","last_active":"2024-04-25T14:02:14.180403+0000","last_peered":"2024-04-25T14:02:14.180403+0000","last_clean":"2024-04-25T14:02:14.180403+0000","last_became_active":"2024-04-25T14:02:08.141389+0000","last_became_peered":"2024-04-25T14:02:08.141389+0000","last_unstale":"2024-04-25T14:02:14.180403+0000","last_undegraded":"2024-04-25T14:02:14.180403+0000","last_fullsized":"2024-04-25T14:02:14.180403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620299+0000","last_change":"2024-04-25T14:02:13.205138+0000","last_active":"2024-04-25T14:02:14.620299+0000","last_peered":"2024-04-25T14:02:14.620299+0000","last_clean":"2024-04-25T14:02:14.620299+0000","last_became_active":"2024-04-25T14:02:13.204712+0000","last_became_peered":"2024-04-25T14:02:13.204712+0000","last_unstale":"2024-04-25T14:02:14.620299+0000","last_undegraded":"2024-04-25T14:02:14.620299+0000","last_fullsized":"2024-04-25T14:02:14.620299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181162+0000","last_change":"2024-04-25T14:02:09.159739+0000","last_active":"2024-04-25T14:02:14.181162+0000","last_peered":"2024-04-25T14:02:14.181162+0000","last_clean":"2024-04-25T14:02:14.181162+0000","last_became_active":"2024-04-25T14:02:09.158156+0000","last_became_peered":"2024-04-25T14:02:09.158156+0000","last_unstale":"2024-04-25T14:02:14.181162+0000","last_undegraded":"2024-04-25T14:02:14.181162+0000","last_fullsized":"2024-04-25T14:02:14.181162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:18.690850+0000","last_change":"2024-04-25T14:02:08.143908+0000","last_active":"2024-04-25T14:02:18.690850+0000","last_peered":"2024-04-25T14:02:18.690850+0000","last_clean":"2024-04-25T14:02:18.690850+0000","last_became_active":"2024-04-25T14:02:08.143478+0000","last_became_peered":"2024-04-25T14:02:08.143478+0000","last_unstale":"2024-04-25T14:02:18.690850+0000","last_undegraded":"2024-04-25T14:02:18.690850+0000","last_fullsized":"2024-04-25T14:02:18.690850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619926+0000","last_change":"2024-04-25T14:02:13.197367+0000","last_active":"2024-04-25T14:02:14.619926+0000","last_peered":"2024-04-25T14:02:14.619926+0000","last_clean":"2024-04-25T14:02:14.619926+0000","last_became_active":"2024-04-25T14:02:13.197029+0000","last_became_peered":"2024-04-25T14:02:13.197029+0000","last_unstale":"2024-04-25T14:02:14.619926+0000","last_undegraded":"2024-04-25T14:02:14.619926+0000","last_fullsized":"2024-04-25T14:02:14.619926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620116+0000","last_change":"2024-04-25T14:02:09.162706+0000","last_active":"2024-04-25T14:02:14.620116+0000","last_peered":"2024-04-25T14:02:14.620116+0000","last_clean":"2024-04-25T14:02:14.620116+0000","last_became_active":"2024-04-25T14:02:09.162316+0000","last_became_peered":"2024-04-25T14:02:09.162316+0000","last_unstale":"2024-04-25T14:02:14.620116+0000","last_undegraded":"2024-04-25T14:02:14.620116+0000","last_fullsized":"2024-04-25T14:02:14.620116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:23.993603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620975+0000","last_change":"2024-04-25T14:02:08.150406+0000","last_active":"2024-04-25T14:02:14.620975+0000","last_peered":"2024-04-25T14:02:14.620975+0000","last_clean":"2024-04-25T14:02:14.620975+0000","last_became_active":"2024-04-25T14:02:08.150057+0000","last_became_peered":"2024-04-25T14:02:08.150057+0000","last_unstale":"2024-04-25T14:02:14.620975+0000","last_undegraded":"2024-04-25T14:02:14.620975+0000","last_fullsized":"2024-04-25T14:02:14.620975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619602+0000","last_change":"2024-04-25T14:02:13.197851+0000","last_active":"2024-04-25T14:02:14.619602+0000","last_peered":"2024-04-25T14:02:14.619602+0000","last_clean":"2024-04-25T14:02:14.619602+0000","last_became_active":"2024-04-25T14:02:13.197109+0000","last_became_peered":"2024-04-25T14:02:13.197109+0000","last_unstale":"2024-04-25T14:02:14.619602+0000","last_undegraded":"2024-04-25T14:02:14.619602+0000","last_fullsized":"2024-04-25T14:02:14.619602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:20.798044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180425+0000","last_change":"2024-04-25T14:02:09.168810+0000","last_active":"2024-04-25T14:02:14.180425+0000","last_peered":"2024-04-25T14:02:14.180425+0000","last_clean":"2024-04-25T14:02:14.180425+0000","last_became_active":"2024-04-25T14:02:09.168323+0000","last_became_peered":"2024-04-25T14:02:09.168323+0000","last_unstale":"2024-04-25T14:02:14.180425+0000","last_undegraded":"2024-04-25T14:02:14.180425+0000","last_fullsized":"2024-04-25T14:02:14.180425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619958+0000","last_change":"2024-04-25T14:02:08.145115+0000","last_active":"2024-04-25T14:02:14.619958+0000","last_peered":"2024-04-25T14:02:14.619958+0000","last_clean":"2024-04-25T14:02:14.619958+0000","last_became_active":"2024-04-25T14:02:08.144853+0000","last_became_peered":"2024-04-25T14:02:08.144853+0000","last_unstale":"2024-04-25T14:02:14.619958+0000","last_undegraded":"2024-04-25T14:02:14.619958+0000","last_fullsized":"2024-04-25T14:02:14.619958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619831+0000","last_change":"2024-04-25T14:02:13.203422+0000","last_active":"2024-04-25T14:02:14.619831+0000","last_peered":"2024-04-25T14:02:14.619831+0000","last_clean":"2024-04-25T14:02:14.619831+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619831+0000","last_undegraded":"2024-04-25T14:02:14.619831+0000","last_fullsized":"2024-04-25T14:02:14.619831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619921+0000","last_change":"2024-04-25T14:02:09.171494+0000","last_active":"2024-04-25T14:02:14.619921+0000","last_peered":"2024-04-25T14:02:14.619921+0000","last_clean":"2024-04-25T14:02:14.619921+0000","last_became_active":"2024-04-25T14:02:09.171163+0000","last_became_peered":"2024-04-25T14:02:09.171163+0000","last_unstale":"2024-04-25T14:02:14.619921+0000","last_undegraded":"2024-04-25T14:02:14.619921+0000","last_fullsized":"2024-04-25T14:02:14.619921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181248+0000","last_change":"2024-04-25T14:02:08.150536+0000","last_active":"2024-04-25T14:02:14.181248+0000","last_peered":"2024-04-25T14:02:14.181248+0000","last_clean":"2024-04-25T14:02:14.181248+0000","last_became_active":"2024-04-25T14:02:08.150156+0000","last_became_peered":"2024-04-25T14:02:08.150156+0000","last_unstale":"2024-04-25T14:02:14.181248+0000","last_undegraded":"2024-04-25T14:02:14.181248+0000","last_fullsized":"2024-04-25T14:02:14.181248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619292+0000","last_change":"2024-04-25T14:02:13.201000+0000","last_active":"2024-04-25T14:02:14.619292+0000","last_peered":"2024-04-25T14:02:14.619292+0000","last_clean":"2024-04-25T14:02:14.619292+0000","last_became_active":"2024-04-25T14:02:13.200594+0000","last_became_peered":"2024-04-25T14:02:13.200594+0000","last_unstale":"2024-04-25T14:02:14.619292+0000","last_undegraded":"2024-04-25T14:02:14.619292+0000","last_fullsized":"2024-04-25T14:02:14.619292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.706373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180405+0000","last_change":"2024-04-25T14:02:13.204286+0000","last_active":"2024-04-25T14:02:14.180405+0000","last_peered":"2024-04-25T14:02:14.180405+0000","last_clean":"2024-04-25T14:02:14.180405+0000","last_became_active":"2024-04-25T14:02:13.203885+0000","last_became_peered":"2024-04-25T14:02:13.203885+0000","last_unstale":"2024-04-25T14:02:14.180405+0000","last_undegraded":"2024-04-25T14:02:14.180405+0000","last_fullsized":"2024-04-25T14:02:14.180405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619363+0000","last_change":"2024-04-25T14:02:09.158647+0000","last_active":"2024-04-25T14:02:14.619363+0000","last_peered":"2024-04-25T14:02:14.619363+0000","last_clean":"2024-04-25T14:02:14.619363+0000","last_became_active":"2024-04-25T14:02:09.158307+0000","last_became_peered":"2024-04-25T14:02:09.158307+0000","last_unstale":"2024-04-25T14:02:14.619363+0000","last_undegraded":"2024-04-25T14:02:14.619363+0000","last_fullsized":"2024-04-25T14:02:14.619363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619239+0000","last_change":"2024-04-25T14:02:08.149002+0000","last_active":"2024-04-25T14:02:14.619239+0000","last_peered":"2024-04-25T14:02:14.619239+0000","last_clean":"2024-04-25T14:02:14.619239+0000","last_became_active":"2024-04-25T14:02:08.148700+0000","last_became_peered":"2024-04-25T14:02:08.148700+0000","last_unstale":"2024-04-25T14:02:14.619239+0000","last_undegraded":"2024-04-25T14:02:14.619239+0000","last_fullsized":"2024-04-25T14:02:14.619239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181096+0000","last_change":"2024-04-25T14:02:13.197649+0000","last_active":"2024-04-25T14:02:14.181096+0000","last_peered":"2024-04-25T14:02:14.181096+0000","last_clean":"2024-04-25T14:02:14.181096+0000","last_became_active":"2024-04-25T14:02:13.197203+0000","last_became_peered":"2024-04-25T14:02:13.197203+0000","last_unstale":"2024-04-25T14:02:14.181096+0000","last_undegraded":"2024-04-25T14:02:14.181096+0000","last_fullsized":"2024-04-25T14:02:14.181096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619786+0000","last_change":"2024-04-25T14:02:09.164726+0000","last_active":"2024-04-25T14:02:14.619786+0000","last_peered":"2024-04-25T14:02:14.619786+0000","last_clean":"2024-04-25T14:02:14.619786+0000","last_became_active":"2024-04-25T14:02:09.164047+0000","last_became_peered":"2024-04-25T14:02:09.164047+0000","last_unstale":"2024-04-25T14:02:14.619786+0000","last_undegraded":"2024-04-25T14:02:14.619786+0000","last_fullsized":"2024-04-25T14:02:14.619786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619586+0000","last_change":"2024-04-25T14:02:08.138596+0000","last_active":"2024-04-25T14:02:14.619586+0000","last_peered":"2024-04-25T14:02:14.619586+0000","last_clean":"2024-04-25T14:02:14.619586+0000","last_became_active":"2024-04-25T14:02:08.138268+0000","last_became_peered":"2024-04-25T14:02:08.138268+0000","last_unstale":"2024-04-25T14:02:14.619586+0000","last_undegraded":"2024-04-25T14:02:14.619586+0000","last_fullsized":"2024-04-25T14:02:14.619586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182061+0000","last_change":"2024-04-25T14:02:08.144180+0000","last_active":"2024-04-25T14:02:14.182061+0000","last_peered":"2024-04-25T14:02:14.182061+0000","last_clean":"2024-04-25T14:02:14.182061+0000","last_became_active":"2024-04-25T14:02:08.142554+0000","last_became_peered":"2024-04-25T14:02:08.142554+0000","last_unstale":"2024-04-25T14:02:14.182061+0000","last_undegraded":"2024-04-25T14:02:14.182061+0000","last_fullsized":"2024-04-25T14:02:14.182061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.163851+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.163561+0000","last_became_peered":"2024-04-25T14:02:09.163561+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620647+0000","last_change":"2024-04-25T14:02:13.200670+0000","last_active":"2024-04-25T14:02:14.620647+0000","last_peered":"2024-04-25T14:02:14.620647+0000","last_clean":"2024-04-25T14:02:14.620647+0000","last_became_active":"2024-04-25T14:02:13.200255+0000","last_became_peered":"2024-04-25T14:02:13.200255+0000","last_unstale":"2024-04-25T14:02:14.620647+0000","last_undegraded":"2024-04-25T14:02:14.620647+0000","last_fullsized":"2024-04-25T14:02:14.620647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:49.451815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620669+0000","last_change":"2024-04-25T14:02:08.136214+0000","last_active":"2024-04-25T14:02:14.620669+0000","last_peered":"2024-04-25T14:02:14.620669+0000","last_clean":"2024-04-25T14:02:14.620669+0000","last_became_active":"2024-04-25T14:02:08.135892+0000","last_became_peered":"2024-04-25T14:02:08.135892+0000","last_unstale":"2024-04-25T14:02:14.620669+0000","last_undegraded":"2024-04-25T14:02:14.620669+0000","last_fullsized":"2024-04-25T14:02:14.620669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:00.331661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.160067+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.159705+0000","last_became_peered":"2024-04-25T14:02:09.159705+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:21.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621148+0000","last_change":"2024-04-25T14:02:13.203327+0000","last_active":"2024-04-25T14:02:14.621148+0000","last_peered":"2024-04-25T14:02:14.621148+0000","last_clean":"2024-04-25T14:02:14.621148+0000","last_became_active":"2024-04-25T14:02:13.202870+0000","last_became_peered":"2024-04-25T14:02:13.202870+0000","last_unstale":"2024-04-25T14:02:14.621148+0000","last_undegraded":"2024-04-25T14:02:14.621148+0000","last_fullsized":"2024-04-25T14:02:14.621148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:39.636544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179770+0000","last_change":"2024-04-25T14:02:09.165255+0000","last_active":"2024-04-25T14:02:14.179770+0000","last_peered":"2024-04-25T14:02:14.179770+0000","last_clean":"2024-04-25T14:02:14.179770+0000","last_became_active":"2024-04-25T14:02:09.164886+0000","last_became_peered":"2024-04-25T14:02:09.164886+0000","last_unstale":"2024-04-25T14:02:14.179770+0000","last_undegraded":"2024-04-25T14:02:14.179770+0000","last_fullsized":"2024-04-25T14:02:14.179770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619161+0000","last_change":"2024-04-25T14:02:09.161880+0000","last_active":"2024-04-25T14:02:14.619161+0000","last_peered":"2024-04-25T14:02:14.619161+0000","last_clean":"2024-04-25T14:02:14.619161+0000","last_became_active":"2024-04-25T14:02:09.161568+0000","last_became_peered":"2024-04-25T14:02:09.161568+0000","last_unstale":"2024-04-25T14:02:14.619161+0000","last_undegraded":"2024-04-25T14:02:14.619161+0000","last_fullsized":"2024-04-25T14:02:14.619161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619443+0000","last_change":"2024-04-25T14:02:09.163237+0000","last_active":"2024-04-25T14:02:14.619443+0000","last_peered":"2024-04-25T14:02:14.619443+0000","last_clean":"2024-04-25T14:02:14.619443+0000","last_became_active":"2024-04-25T14:02:09.162870+0000","last_became_peered":"2024-04-25T14:02:09.162870+0000","last_unstale":"2024-04-25T14:02:14.619443+0000","last_undegraded":"2024-04-25T14:02:14.619443+0000","last_fullsized":"2024-04-25T14:02:14.619443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180151+0000","last_change":"2024-04-25T14:02:09.167689+0000","last_active":"2024-04-25T14:02:14.180151+0000","last_peered":"2024-04-25T14:02:14.180151+0000","last_clean":"2024-04-25T14:02:14.180151+0000","last_became_active":"2024-04-25T14:02:09.167379+0000","last_became_peered":"2024-04-25T14:02:09.167379+0000","last_unstale":"2024-04-25T14:02:14.180151+0000","last_undegraded":"2024-04-25T14:02:14.180151+0000","last_fullsized":"2024-04-25T14:02:14.180151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621304+0000","last_change":"2024-04-25T14:02:09.155156+0000","last_active":"2024-04-25T14:02:14.621304+0000","last_peered":"2024-04-25T14:02:14.621304+0000","last_clean":"2024-04-25T14:02:14.621304+0000","last_became_active":"2024-04-25T14:02:09.154497+0000","last_became_peered":"2024-04-25T14:02:09.154497+0000","last_unstale":"2024-04-25T14:02:14.621304+0000","last_undegraded":"2024-04-25T14:02:14.621304+0000","last_fullsized":"2024-04-25T14:02:14.621304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620178+0000","last_change":"2024-04-25T14:02:09.161627+0000","last_active":"2024-04-25T14:02:14.620178+0000","last_peered":"2024-04-25T14:02:14.620178+0000","last_clean":"2024-04-25T14:02:14.620178+0000","last_became_active":"2024-04-25T14:02:09.161321+0000","last_became_peered":"2024-04-25T14:02:09.161321+0000","last_unstale":"2024-04-25T14:02:14.620178+0000","last_undegraded":"2024-04-25T14:02:14.620178+0000","last_fullsized":"2024-04-25T14:02:14.620178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181400+0000","last_change":"2024-04-25T14:02:09.177833+0000","last_active":"2024-04-25T14:02:14.181400+0000","last_peered":"2024-04-25T14:02:14.181400+0000","last_clean":"2024-04-25T14:02:14.181400+0000","last_became_active":"2024-04-25T14:02:09.177530+0000","last_became_peered":"2024-04-25T14:02:09.177530+0000","last_unstale":"2024-04-25T14:02:14.181400+0000","last_undegraded":"2024-04-25T14:02:14.181400+0000","last_fullsized":"2024-04-25T14:02:14.181400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621026+0000","last_change":"2024-04-25T14:02:09.160341+0000","last_active":"2024-04-25T14:02:14.621026+0000","last_peered":"2024-04-25T14:02:14.621026+0000","last_clean":"2024-04-25T14:02:14.621026+0000","last_became_active":"2024-04-25T14:02:09.160028+0000","last_became_peered":"2024-04-25T14:02:09.160028+0000","last_unstale":"2024-04-25T14:02:14.621026+0000","last_undegraded":"2024-04-25T14:02:14.621026+0000","last_fullsized":"2024-04-25T14:02:14.621026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619081+0000","last_change":"2024-04-25T14:02:09.163936+0000","last_active":"2024-04-25T14:02:14.619081+0000","last_peered":"2024-04-25T14:02:14.619081+0000","last_clean":"2024-04-25T14:02:14.619081+0000","last_became_active":"2024-04-25T14:02:09.163303+0000","last_became_peered":"2024-04-25T14:02:09.163303+0000","last_unstale":"2024-04-25T14:02:14.619081+0000","last_undegraded":"2024-04-25T14:02:14.619081+0000","last_fullsized":"2024-04-25T14:02:14.619081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619413+0000","last_change":"2024-04-25T14:02:09.154839+0000","last_active":"2024-04-25T14:02:14.619413+0000","last_peered":"2024-04-25T14:02:14.619413+0000","last_clean":"2024-04-25T14:02:14.619413+0000","last_became_active":"2024-04-25T14:02:09.154443+0000","last_became_peered":"2024-04-25T14:02:09.154443+0000","last_unstale":"2024-04-25T14:02:14.619413+0000","last_undegraded":"2024-04-25T14:02:14.619413+0000","last_fullsized":"2024-04-25T14:02:14.619413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619286+0000","last_change":"2024-04-25T14:02:09.171782+0000","last_active":"2024-04-25T14:02:14.619286+0000","last_peered":"2024-04-25T14:02:14.619286+0000","last_clean":"2024-04-25T14:02:14.619286+0000","last_became_active":"2024-04-25T14:02:09.171158+0000","last_became_peered":"2024-04-25T14:02:09.171158+0000","last_unstale":"2024-04-25T14:02:14.619286+0000","last_undegraded":"2024-04-25T14:02:14.619286+0000","last_fullsized":"2024-04-25T14:02:14.619286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179956+0000","last_change":"2024-04-25T14:02:09.177793+0000","last_active":"2024-04-25T14:02:14.179956+0000","last_peered":"2024-04-25T14:02:14.179956+0000","last_clean":"2024-04-25T14:02:14.179956+0000","last_became_active":"2024-04-25T14:02:09.177475+0000","last_became_peered":"2024-04-25T14:02:09.177475+0000","last_unstale":"2024-04-25T14:02:14.179956+0000","last_undegraded":"2024-04-25T14:02:14.179956+0000","last_fullsized":"2024-04-25T14:02:14.179956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181451+0000","last_change":"2024-04-25T14:02:09.166782+0000","last_active":"2024-04-25T14:02:14.181451+0000","last_peered":"2024-04-25T14:02:14.181451+0000","last_clean":"2024-04-25T14:02:14.181451+0000","last_became_active":"2024-04-25T14:02:09.166277+0000","last_became_peered":"2024-04-25T14:02:09.166277+0000","last_unstale":"2024-04-25T14:02:14.181451+0000","last_undegraded":"2024-04-25T14:02:14.181451+0000","last_fullsized":"2024-04-25T14:02:14.181451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181472+0000","last_change":"2024-04-25T14:02:09.167448+0000","last_active":"2024-04-25T14:02:14.181472+0000","last_peered":"2024-04-25T14:02:14.181472+0000","last_clean":"2024-04-25T14:02:14.181472+0000","last_became_active":"2024-04-25T14:02:09.167136+0000","last_became_peered":"2024-04-25T14:02:09.167136+0000","last_unstale":"2024-04-25T14:02:14.181472+0000","last_undegraded":"2024-04-25T14:02:14.181472+0000","last_fullsized":"2024-04-25T14:02:14.181472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620713+0000","last_change":"2024-04-25T14:02:09.167572+0000","last_active":"2024-04-25T14:02:14.620713+0000","last_peered":"2024-04-25T14:02:14.620713+0000","last_clean":"2024-04-25T14:02:14.620713+0000","last_became_active":"2024-04-25T14:02:09.167227+0000","last_became_peered":"2024-04-25T14:02:09.167227+0000","last_unstale":"2024-04-25T14:02:14.620713+0000","last_undegraded":"2024-04-25T14:02:14.620713+0000","last_fullsized":"2024-04-25T14:02:14.620713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620673+0000","last_change":"2024-04-25T14:02:09.174321+0000","last_active":"2024-04-25T14:02:14.620673+0000","last_peered":"2024-04-25T14:02:14.620673+0000","last_clean":"2024-04-25T14:02:14.620673+0000","last_became_active":"2024-04-25T14:02:09.174015+0000","last_became_peered":"2024-04-25T14:02:09.174015+0000","last_unstale":"2024-04-25T14:02:14.620673+0000","last_undegraded":"2024-04-25T14:02:14.620673+0000","last_fullsized":"2024-04-25T14:02:14.620673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618957+0000","last_change":"2024-04-25T14:02:09.169554+0000","last_active":"2024-04-25T14:02:14.618957+0000","last_peered":"2024-04-25T14:02:14.618957+0000","last_clean":"2024-04-25T14:02:14.618957+0000","last_became_active":"2024-04-25T14:02:09.168555+0000","last_became_peered":"2024-04-25T14:02:09.168555+0000","last_unstale":"2024-04-25T14:02:14.618957+0000","last_undegraded":"2024-04-25T14:02:14.618957+0000","last_fullsized":"2024-04-25T14:02:14.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619417+0000","last_change":"2024-04-25T14:02:09.160170+0000","last_active":"2024-04-25T14:02:14.619417+0000","last_peered":"2024-04-25T14:02:14.619417+0000","last_clean":"2024-04-25T14:02:14.619417+0000","last_became_active":"2024-04-25T14:02:09.159871+0000","last_became_peered":"2024-04-25T14:02:09.159871+0000","last_unstale":"2024-04-25T14:02:14.619417+0000","last_undegraded":"2024-04-25T14:02:14.619417+0000","last_fullsized":"2024-04-25T14:02:14.619417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179980+0000","last_change":"2024-04-25T14:02:09.159887+0000","last_active":"2024-04-25T14:02:14.179980+0000","last_peered":"2024-04-25T14:02:14.179980+0000","last_clean":"2024-04-25T14:02:14.179980+0000","last_became_active":"2024-04-25T14:02:09.159431+0000","last_became_peered":"2024-04-25T14:02:09.159431+0000","last_unstale":"2024-04-25T14:02:14.179980+0000","last_undegraded":"2024-04-25T14:02:14.179980+0000","last_fullsized":"2024-04-25T14:02:14.179980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620557+0000","last_change":"2024-04-25T14:02:09.167290+0000","last_active":"2024-04-25T14:02:14.620557+0000","last_peered":"2024-04-25T14:02:14.620557+0000","last_clean":"2024-04-25T14:02:14.620557+0000","last_became_active":"2024-04-25T14:02:09.166974+0000","last_became_peered":"2024-04-25T14:02:09.166974+0000","last_unstale":"2024-04-25T14:02:14.620557+0000","last_undegraded":"2024-04-25T14:02:14.620557+0000","last_fullsized":"2024-04-25T14:02:14.620557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620085+0000","last_change":"2024-04-25T14:02:09.169185+0000","last_active":"2024-04-25T14:02:14.620085+0000","last_peered":"2024-04-25T14:02:14.620085+0000","last_clean":"2024-04-25T14:02:14.620085+0000","last_became_active":"2024-04-25T14:02:09.168862+0000","last_became_peered":"2024-04-25T14:02:09.168862+0000","last_unstale":"2024-04-25T14:02:14.620085+0000","last_undegraded":"2024-04-25T14:02:14.620085+0000","last_fullsized":"2024-04-25T14:02:14.620085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180614+0000","last_change":"2024-04-25T14:02:09.174006+0000","last_active":"2024-04-25T14:02:14.180614+0000","last_peered":"2024-04-25T14:02:14.180614+0000","last_clean":"2024-04-25T14:02:14.180614+0000","last_became_active":"2024-04-25T14:02:09.173662+0000","last_became_peered":"2024-04-25T14:02:09.173662+0000","last_unstale":"2024-04-25T14:02:14.180614+0000","last_undegraded":"2024-04-25T14:02:14.180614+0000","last_fullsized":"2024-04-25T14:02:14.180614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181263+0000","last_change":"2024-04-25T14:02:09.174331+0000","last_active":"2024-04-25T14:02:14.181263+0000","last_peered":"2024-04-25T14:02:14.181263+0000","last_clean":"2024-04-25T14:02:14.181263+0000","last_became_active":"2024-04-25T14:02:09.173998+0000","last_became_peered":"2024-04-25T14:02:09.173998+0000","last_unstale":"2024-04-25T14:02:14.181263+0000","last_undegraded":"2024-04-25T14:02:14.181263+0000","last_fullsized":"2024-04-25T14:02:14.181263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182263+0000","last_change":"2024-04-25T14:02:09.166767+0000","last_active":"2024-04-25T14:02:14.182263+0000","last_peered":"2024-04-25T14:02:14.182263+0000","last_clean":"2024-04-25T14:02:14.182263+0000","last_became_active":"2024-04-25T14:02:09.165131+0000","last_became_peered":"2024-04-25T14:02:09.165131+0000","last_unstale":"2024-04-25T14:02:14.182263+0000","last_undegraded":"2024-04-25T14:02:14.182263+0000","last_fullsized":"2024-04-25T14:02:14.182263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:07.386855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619681+0000","last_change":"2024-04-25T14:02:09.164960+0000","last_active":"2024-04-25T14:02:14.619681+0000","last_peered":"2024-04-25T14:02:14.619681+0000","last_clean":"2024-04-25T14:02:14.619681+0000","last_became_active":"2024-04-25T14:02:09.164417+0000","last_became_peered":"2024-04-25T14:02:09.164417+0000","last_unstale":"2024-04-25T14:02:14.619681+0000","last_undegraded":"2024-04-25T14:02:14.619681+0000","last_fullsized":"2024-04-25T14:02:14.619681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180423+0000","last_change":"2024-04-25T14:02:09.161177+0000","last_active":"2024-04-25T14:02:14.180423+0000","last_peered":"2024-04-25T14:02:14.180423+0000","last_clean":"2024-04-25T14:02:14.180423+0000","last_became_active":"2024-04-25T14:02:09.160404+0000","last_became_peered":"2024-04-25T14:02:09.160404+0000","last_unstale":"2024-04-25T14:02:14.180423+0000","last_undegraded":"2024-04-25T14:02:14.180423+0000","last_fullsized":"2024-04-25T14:02:14.180423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619241+0000","last_change":"2024-04-25T14:02:09.170498+0000","last_active":"2024-04-25T14:02:14.619241+0000","last_peered":"2024-04-25T14:02:14.619241+0000","last_clean":"2024-04-25T14:02:14.619241+0000","last_became_active":"2024-04-25T14:02:09.167994+0000","last_became_peered":"2024-04-25T14:02:09.167994+0000","last_unstale":"2024-04-25T14:02:14.619241+0000","last_undegraded":"2024-04-25T14:02:14.619241+0000","last_fullsized":"2024-04-25T14:02:14.619241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180565+0000","last_change":"2024-04-25T14:02:09.155470+0000","last_active":"2024-04-25T14:02:14.180565+0000","last_peered":"2024-04-25T14:02:14.180565+0000","last_clean":"2024-04-25T14:02:14.180565+0000","last_became_active":"2024-04-25T14:02:09.155123+0000","last_became_peered":"2024-04-25T14:02:09.155123+0000","last_unstale":"2024-04-25T14:02:14.180565+0000","last_undegraded":"2024-04-25T14:02:14.180565+0000","last_fullsized":"2024-04-25T14:02:14.180565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619577+0000","last_change":"2024-04-25T14:02:09.163082+0000","last_active":"2024-04-25T14:02:14.619577+0000","last_peered":"2024-04-25T14:02:14.619577+0000","last_clean":"2024-04-25T14:02:14.619577+0000","last_became_active":"2024-04-25T14:02:09.162148+0000","last_became_peered":"2024-04-25T14:02:09.162148+0000","last_unstale":"2024-04-25T14:02:14.619577+0000","last_undegraded":"2024-04-25T14:02:14.619577+0000","last_fullsized":"2024-04-25T14:02:14.619577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.167223+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.166924+0000","last_became_peered":"2024-04-25T14:02:09.166924+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619169+0000","last_change":"2024-04-25T14:02:09.159838+0000","last_active":"2024-04-25T14:02:14.619169+0000","last_peered":"2024-04-25T14:02:14.619169+0000","last_clean":"2024-04-25T14:02:14.619169+0000","last_became_active":"2024-04-25T14:02:09.159551+0000","last_became_peered":"2024-04-25T14:02:09.159551+0000","last_unstale":"2024-04-25T14:02:14.619169+0000","last_undegraded":"2024-04-25T14:02:14.619169+0000","last_fullsized":"2024-04-25T14:02:14.619169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182085+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.182085+0000","last_peered":"2024-04-25T14:02:14.182085+0000","last_clean":"2024-04-25T14:02:14.182085+0000","last_became_active":"2024-04-25T14:02:09.168711+0000","last_became_peered":"2024-04-25T14:02:09.168711+0000","last_unstale":"2024-04-25T14:02:14.182085+0000","last_undegraded":"2024-04-25T14:02:14.182085+0000","last_fullsized":"2024-04-25T14:02:14.182085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:02:36.833 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:02:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.975+0000 7ff80812e640 1 Processor -- start 2024-04-25T14:02:36.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.975+0000 7ff80812e640 1 -- start start 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff80812e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff800162790 0x7ff8000b53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff80812e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8000b5d00 con 0x7ff8000b58f0 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff80812e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff800161cd0 con 0x7ff8000d2870 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff80812e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff800162300 con 0x7ff800162790 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff805eca640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff800162790 0x7ff8000b53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff805eca640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff800162790 0x7ff8000b53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38648/0 (socket says 172.21.15.150:38648) 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.976+0000 7ff8056c9640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff8000b58f0 0x7ff8000d2160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55662/0 (socket says 172.21.15.150:55662) 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.977+0000 7ff805eca640 1 -- 172.21.15.150:0/3327130379 learned_addr learned my addr 172.21.15.150:0/3327130379 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.977+0000 7ff805eca640 1 -- 172.21.15.150:0/3327130379 >> v1:172.21.15.187:6789/0 conn(0x7ff8000d2870 legacy=0x7ff80015dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.977+0000 7ff805eca640 1 -- 172.21.15.150:0/3327130379 >> v1:172.21.15.150:6789/0 conn(0x7ff8000b58f0 legacy=0x7ff8000d2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.977+0000 7ff805eca640 1 -- 172.21.15.150:0/3327130379 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80015f300 con 0x7ff800162790 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.977+0000 7ff805eca640 1 --2- 172.21.15.150:0/3327130379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff800162790 0x7ff8000b53b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff800162480 tx=0x7ff7fc02a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e284784cc3dda619 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff804ec8640 1 -- 172.21.15.150:0/3327130379 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc037070 con 0x7ff800162790 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff804ec8640 1 -- 172.21.15.150:0/3327130379 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7fc032440 con 0x7ff800162790 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff804ec8640 1 -- 172.21.15.150:0/3327130379 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc032810 con 0x7ff800162790 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 -- 172.21.15.150:0/3327130379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff800162790 msgr2=0x7ff8000b53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 --2- 172.21.15.150:0/3327130379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff800162790 0x7ff8000b53b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff800162480 tx=0x7ff7fc02a2c0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 -- 172.21.15.150:0/3327130379 shutdown_connections 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 --2- 172.21.15.150:0/3327130379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff800162790 0x7ff8000b53b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 -- 172.21.15.150:0/3327130379 >> 172.21.15.150:0/3327130379 conn(0x7ff8000d03b0 msgr2=0x7ff8000d07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:36.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 -- 172.21.15.150:0/3327130379 shutdown_connections 2024-04-25T14:02:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.978+0000 7ff80812e640 1 -- 172.21.15.150:0/3327130379 wait complete. 2024-04-25T14:02:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.980+0000 7ff80812e640 1 Processor -- start 2024-04-25T14:02:36.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.980+0000 7ff80812e640 1 -- start start 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d2870 0x7ff8000c9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 0x7ff8000c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff8000d2e00 con 0x7ff8000b58f0 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff8000d2c80 con 0x7ff800162790 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.981+0000 7ff80812e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff80015e5f0 con 0x7ff8000d2870 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38656/0 (socket says 172.21.15.150:38656) 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 -- 172.21.15.150:0/152884160 learned_addr learned my addr 172.21.15.150:0/152884160 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff8056c9640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d2870 0x7ff8000c9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff8066cb640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 0x7ff8000c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 -- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d2870 msgr2=0x7ff8000c9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d2870 0x7ff8000c9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 -- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 msgr2=0x7ff8000c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 0x7ff8000c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 -- 172.21.15.150:0/152884160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff80006e6f0 con 0x7ff8000b58f0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff8066cb640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 0x7ff8000c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff805eca640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009870 tx=0x7ff7fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc037040 con 0x7ff8000b58f0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7fc004230 con 0x7ff8000b58f0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff80006e9d0 con 0x7ff8000b58f0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc03c730 con 0x7ff8000b58f0 2024-04-25T14:02:36.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.982+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff80015d410 con 0x7ff8000b58f0 2024-04-25T14:02:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.983+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8001f6b10 con 0x7ff8000b58f0 2024-04-25T14:02:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.984+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff7fc03a070 con 0x7ff8000b58f0 2024-04-25T14:02:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.984+0000 7ff7f67fc640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 0x7ff7e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:36.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.984+0000 7ff8056c9640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 0x7ff7e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.985+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ff7fc0b5e00 con 0x7ff8000b58f0 2024-04-25T14:02:36.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.985+0000 7ff8056c9640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 0x7ff7e40769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff8000ce0e0 tx=0x7ff7f0007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:36.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:36.989+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7fc041020 con 0x7ff8000b58f0 2024-04-25T14:02:37.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.158+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff800076a40 con 0x7ff7e40744b0 2024-04-25T14:02:37.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.166+0000 7ff7f67fc640 1 -- 172.21.15.150:0/152884160 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332522 (secure 0 0 0) 0x7ff800076a40 con 0x7ff7e40744b0 2024-04-25T14:02:37.168 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:37.170 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 msgr2=0x7ff7e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 0x7ff7e40769c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff8000ce0e0 tx=0x7ff7f0007e80 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 msgr2=0x7ff80015ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009870 tx=0x7ff7fc0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 shutdown_connections 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff7e40744b0 0x7ff7e40769c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff800162790 0x7ff8000c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8000d2870 0x7ff8000c9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 --2- 172.21.15.150:0/152884160 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8000b58f0 0x7ff80015ced0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 >> 172.21.15.150:0/152884160 conn(0x7ff8000d03b0 msgr2=0x7ff800160d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:37.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 shutdown_connections 2024-04-25T14:02:37.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.170+0000 7ff80812e640 1 -- 172.21.15.150:0/152884160 wait complete. 2024-04-25T14:02:37.180 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-25T14:02:36.592863+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":149,"num_write_kb":1646,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219392,"kb_used_data":3208,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755328,"statfs":{"total":773094113280,"available":772869455872,"internally_reserved":0,"allocated":3284992,"data_stored":1992064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003244"},"pg_stats":[{"pgid":"2.1f","version":"19'4","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180578+0000","last_change":"2024-04-25T14:02:08.147737+0000","last_active":"2024-04-25T14:02:14.180578+0000","last_peered":"2024-04-25T14:02:14.180578+0000","last_clean":"2024-04-25T14:02:14.180578+0000","last_became_active":"2024-04-25T14:02:08.147072+0000","last_became_peered":"2024-04-25T14:02:08.147072+0000","last_unstale":"2024-04-25T14:02:14.180578+0000","last_undegraded":"2024-04-25T14:02:14.180578+0000","last_fullsized":"2024-04-25T14:02:14.180578+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621342+0000","last_change":"2024-04-25T14:02:09.168014+0000","last_active":"2024-04-25T14:02:14.621342+0000","last_peered":"2024-04-25T14:02:14.621342+0000","last_clean":"2024-04-25T14:02:14.621342+0000","last_became_active":"2024-04-25T14:02:09.166846+0000","last_became_peered":"2024-04-25T14:02:09.166846+0000","last_unstale":"2024-04-25T14:02:14.621342+0000","last_undegraded":"2024-04-25T14:02:14.621342+0000","last_fullsized":"2024-04-25T14:02:14.621342+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619101+0000","last_change":"2024-04-25T14:02:13.203189+0000","last_active":"2024-04-25T14:02:14.619101+0000","last_peered":"2024-04-25T14:02:14.619101+0000","last_clean":"2024-04-25T14:02:14.619101+0000","last_became_active":"2024-04-25T14:02:13.201743+0000","last_became_peered":"2024-04-25T14:02:13.201743+0000","last_unstale":"2024-04-25T14:02:14.619101+0000","last_undegraded":"2024-04-25T14:02:14.619101+0000","last_fullsized":"2024-04-25T14:02:14.619101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621459+0000","last_change":"2024-04-25T14:02:08.147961+0000","last_active":"2024-04-25T14:02:14.621459+0000","last_peered":"2024-04-25T14:02:14.621459+0000","last_clean":"2024-04-25T14:02:14.621459+0000","last_became_active":"2024-04-25T14:02:08.147652+0000","last_became_peered":"2024-04-25T14:02:08.147652+0000","last_unstale":"2024-04-25T14:02:14.621459+0000","last_undegraded":"2024-04-25T14:02:14.621459+0000","last_fullsized":"2024-04-25T14:02:14.621459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180307+0000","last_change":"2024-04-25T14:02:09.173887+0000","last_active":"2024-04-25T14:02:14.180307+0000","last_peered":"2024-04-25T14:02:14.180307+0000","last_clean":"2024-04-25T14:02:14.180307+0000","last_became_active":"2024-04-25T14:02:09.173583+0000","last_became_peered":"2024-04-25T14:02:09.173583+0000","last_unstale":"2024-04-25T14:02:14.180307+0000","last_undegraded":"2024-04-25T14:02:14.180307+0000","last_fullsized":"2024-04-25T14:02:14.180307+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619293+0000","last_change":"2024-04-25T14:02:13.202095+0000","last_active":"2024-04-25T14:02:14.619293+0000","last_peered":"2024-04-25T14:02:14.619293+0000","last_clean":"2024-04-25T14:02:14.619293+0000","last_became_active":"2024-04-25T14:02:13.201620+0000","last_became_peered":"2024-04-25T14:02:13.201620+0000","last_unstale":"2024-04-25T14:02:14.619293+0000","last_undegraded":"2024-04-25T14:02:14.619293+0000","last_fullsized":"2024-04-25T14:02:14.619293+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:25.777824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181439+0000","last_change":"2024-04-25T14:02:08.139279+0000","last_active":"2024-04-25T14:02:14.181439+0000","last_peered":"2024-04-25T14:02:14.181439+0000","last_clean":"2024-04-25T14:02:14.181439+0000","last_became_active":"2024-04-25T14:02:08.138923+0000","last_became_peered":"2024-04-25T14:02:08.138923+0000","last_unstale":"2024-04-25T14:02:14.181439+0000","last_undegraded":"2024-04-25T14:02:14.181439+0000","last_fullsized":"2024-04-25T14:02:14.181439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:22.987870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619690+0000","last_change":"2024-04-25T14:02:09.162551+0000","last_active":"2024-04-25T14:02:14.619690+0000","last_peered":"2024-04-25T14:02:14.619690+0000","last_clean":"2024-04-25T14:02:14.619690+0000","last_became_active":"2024-04-25T14:02:09.161794+0000","last_became_peered":"2024-04-25T14:02:09.161794+0000","last_unstale":"2024-04-25T14:02:14.619690+0000","last_undegraded":"2024-04-25T14:02:14.619690+0000","last_fullsized":"2024-04-25T14:02:14.619690+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619746+0000","last_change":"2024-04-25T14:02:13.203879+0000","last_active":"2024-04-25T14:02:14.619746+0000","last_peered":"2024-04-25T14:02:14.619746+0000","last_clean":"2024-04-25T14:02:14.619746+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619746+0000","last_undegraded":"2024-04-25T14:02:14.619746+0000","last_fullsized":"2024-04-25T14:02:14.619746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:34.794247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180468+0000","last_change":"2024-04-25T14:02:08.148407+0000","last_active":"2024-04-25T14:02:14.180468+0000","last_peered":"2024-04-25T14:02:14.180468+0000","last_clean":"2024-04-25T14:02:14.180468+0000","last_became_active":"2024-04-25T14:02:08.148049+0000","last_became_peered":"2024-04-25T14:02:08.148049+0000","last_unstale":"2024-04-25T14:02:14.180468+0000","last_undegraded":"2024-04-25T14:02:14.180468+0000","last_fullsized":"2024-04-25T14:02:14.180468+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620378+0000","last_change":"2024-04-25T14:02:09.162054+0000","last_active":"2024-04-25T14:02:14.620378+0000","last_peered":"2024-04-25T14:02:14.620378+0000","last_clean":"2024-04-25T14:02:14.620378+0000","last_became_active":"2024-04-25T14:02:09.161561+0000","last_became_peered":"2024-04-25T14:02:09.161561+0000","last_unstale":"2024-04-25T14:02:14.620378+0000","last_undegraded":"2024-04-25T14:02:14.620378+0000","last_fullsized":"2024-04-25T14:02:14.620378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620224+0000","last_change":"2024-04-25T14:02:13.202231+0000","last_active":"2024-04-25T14:02:14.620224+0000","last_peered":"2024-04-25T14:02:14.620224+0000","last_clean":"2024-04-25T14:02:14.620224+0000","last_became_active":"2024-04-25T14:02:13.201805+0000","last_became_peered":"2024-04-25T14:02:13.201805+0000","last_unstale":"2024-04-25T14:02:14.620224+0000","last_undegraded":"2024-04-25T14:02:14.620224+0000","last_fullsized":"2024-04-25T14:02:14.620224+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:12.379500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181919+0000","last_change":"2024-04-25T14:02:09.164630+0000","last_active":"2024-04-25T14:02:14.181919+0000","last_peered":"2024-04-25T14:02:14.181919+0000","last_clean":"2024-04-25T14:02:14.181919+0000","last_became_active":"2024-04-25T14:02:09.164292+0000","last_became_peered":"2024-04-25T14:02:09.164292+0000","last_unstale":"2024-04-25T14:02:14.181919+0000","last_undegraded":"2024-04-25T14:02:14.181919+0000","last_fullsized":"2024-04-25T14:02:14.181919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180969+0000","last_change":"2024-04-25T14:02:08.136088+0000","last_active":"2024-04-25T14:02:14.180969+0000","last_peered":"2024-04-25T14:02:14.180969+0000","last_clean":"2024-04-25T14:02:14.180969+0000","last_became_active":"2024-04-25T14:02:08.135716+0000","last_became_peered":"2024-04-25T14:02:08.135716+0000","last_unstale":"2024-04-25T14:02:14.180969+0000","last_undegraded":"2024-04-25T14:02:14.180969+0000","last_fullsized":"2024-04-25T14:02:14.180969+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:12:27.040903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620573+0000","last_change":"2024-04-25T14:02:13.196592+0000","last_active":"2024-04-25T14:02:14.620573+0000","last_peered":"2024-04-25T14:02:14.620573+0000","last_clean":"2024-04-25T14:02:14.620573+0000","last_became_active":"2024-04-25T14:02:13.196120+0000","last_became_peered":"2024-04-25T14:02:13.196120+0000","last_unstale":"2024-04-25T14:02:14.620573+0000","last_undegraded":"2024-04-25T14:02:14.620573+0000","last_fullsized":"2024-04-25T14:02:14.620573+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620568+0000","last_change":"2024-04-25T14:02:09.163625+0000","last_active":"2024-04-25T14:02:14.620568+0000","last_peered":"2024-04-25T14:02:14.620568+0000","last_clean":"2024-04-25T14:02:14.620568+0000","last_became_active":"2024-04-25T14:02:09.163320+0000","last_became_peered":"2024-04-25T14:02:09.163320+0000","last_unstale":"2024-04-25T14:02:14.620568+0000","last_undegraded":"2024-04-25T14:02:14.620568+0000","last_fullsized":"2024-04-25T14:02:14.620568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619947+0000","last_change":"2024-04-25T14:02:08.140552+0000","last_active":"2024-04-25T14:02:14.619947+0000","last_peered":"2024-04-25T14:02:14.619947+0000","last_clean":"2024-04-25T14:02:14.619947+0000","last_became_active":"2024-04-25T14:02:08.139997+0000","last_became_peered":"2024-04-25T14:02:08.139997+0000","last_unstale":"2024-04-25T14:02:14.619947+0000","last_undegraded":"2024-04-25T14:02:14.619947+0000","last_fullsized":"2024-04-25T14:02:14.619947+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620055+0000","last_change":"2024-04-25T14:02:13.204879+0000","last_active":"2024-04-25T14:02:14.620055+0000","last_peered":"2024-04-25T14:02:14.620055+0000","last_clean":"2024-04-25T14:02:14.620055+0000","last_became_active":"2024-04-25T14:02:13.203934+0000","last_became_peered":"2024-04-25T14:02:13.203934+0000","last_unstale":"2024-04-25T14:02:14.620055+0000","last_undegraded":"2024-04-25T14:02:14.620055+0000","last_fullsized":"2024-04-25T14:02:14.620055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619890+0000","last_change":"2024-04-25T14:02:09.157409+0000","last_active":"2024-04-25T14:02:14.619890+0000","last_peered":"2024-04-25T14:02:14.619890+0000","last_clean":"2024-04-25T14:02:14.619890+0000","last_became_active":"2024-04-25T14:02:09.157102+0000","last_became_peered":"2024-04-25T14:02:09.157102+0000","last_unstale":"2024-04-25T14:02:14.619890+0000","last_undegraded":"2024-04-25T14:02:14.619890+0000","last_fullsized":"2024-04-25T14:02:14.619890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619463+0000","last_change":"2024-04-25T14:02:08.138257+0000","last_active":"2024-04-25T14:02:14.619463+0000","last_peered":"2024-04-25T14:02:14.619463+0000","last_clean":"2024-04-25T14:02:14.619463+0000","last_became_active":"2024-04-25T14:02:08.137925+0000","last_became_peered":"2024-04-25T14:02:08.137925+0000","last_unstale":"2024-04-25T14:02:14.619463+0000","last_undegraded":"2024-04-25T14:02:14.619463+0000","last_fullsized":"2024-04-25T14:02:14.619463+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619999+0000","last_change":"2024-04-25T14:02:13.204203+0000","last_active":"2024-04-25T14:02:14.619999+0000","last_peered":"2024-04-25T14:02:14.619999+0000","last_clean":"2024-04-25T14:02:14.619999+0000","last_became_active":"2024-04-25T14:02:13.203578+0000","last_became_peered":"2024-04-25T14:02:13.203578+0000","last_unstale":"2024-04-25T14:02:14.619999+0000","last_undegraded":"2024-04-25T14:02:14.619999+0000","last_fullsized":"2024-04-25T14:02:14.619999+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620454+0000","last_change":"2024-04-25T14:02:09.153035+0000","last_active":"2024-04-25T14:02:14.620454+0000","last_peered":"2024-04-25T14:02:14.620454+0000","last_clean":"2024-04-25T14:02:14.620454+0000","last_became_active":"2024-04-25T14:02:09.152071+0000","last_became_peered":"2024-04-25T14:02:09.152071+0000","last_unstale":"2024-04-25T14:02:14.620454+0000","last_undegraded":"2024-04-25T14:02:14.620454+0000","last_fullsized":"2024-04-25T14:02:14.620454+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:11.100888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180075+0000","last_change":"2024-04-25T14:02:08.141740+0000","last_active":"2024-04-25T14:02:14.180075+0000","last_peered":"2024-04-25T14:02:14.180075+0000","last_clean":"2024-04-25T14:02:14.180075+0000","last_became_active":"2024-04-25T14:02:08.141400+0000","last_became_peered":"2024-04-25T14:02:08.141400+0000","last_unstale":"2024-04-25T14:02:14.180075+0000","last_undegraded":"2024-04-25T14:02:14.180075+0000","last_fullsized":"2024-04-25T14:02:14.180075+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619288+0000","last_change":"2024-04-25T14:02:13.202697+0000","last_active":"2024-04-25T14:02:14.619288+0000","last_peered":"2024-04-25T14:02:14.619288+0000","last_clean":"2024-04-25T14:02:14.619288+0000","last_became_active":"2024-04-25T14:02:13.201555+0000","last_became_peered":"2024-04-25T14:02:13.201555+0000","last_unstale":"2024-04-25T14:02:14.619288+0000","last_undegraded":"2024-04-25T14:02:14.619288+0000","last_fullsized":"2024-04-25T14:02:14.619288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:43.769087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180683+0000","last_change":"2024-04-25T14:02:09.166496+0000","last_active":"2024-04-25T14:02:14.180683+0000","last_peered":"2024-04-25T14:02:14.180683+0000","last_clean":"2024-04-25T14:02:14.180683+0000","last_became_active":"2024-04-25T14:02:09.166180+0000","last_became_peered":"2024-04-25T14:02:09.166180+0000","last_unstale":"2024-04-25T14:02:14.180683+0000","last_undegraded":"2024-04-25T14:02:14.180683+0000","last_fullsized":"2024-04-25T14:02:14.180683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181386+0000","last_change":"2024-04-25T14:02:08.143988+0000","last_active":"2024-04-25T14:02:14.181386+0000","last_peered":"2024-04-25T14:02:14.181386+0000","last_clean":"2024-04-25T14:02:14.181386+0000","last_became_active":"2024-04-25T14:02:08.143639+0000","last_became_peered":"2024-04-25T14:02:08.143639+0000","last_unstale":"2024-04-25T14:02:14.181386+0000","last_undegraded":"2024-04-25T14:02:14.181386+0000","last_fullsized":"2024-04-25T14:02:14.181386+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620239+0000","last_change":"2024-04-25T14:02:13.193130+0000","last_active":"2024-04-25T14:02:14.620239+0000","last_peered":"2024-04-25T14:02:14.620239+0000","last_clean":"2024-04-25T14:02:14.620239+0000","last_became_active":"2024-04-25T14:02:13.192664+0000","last_became_peered":"2024-04-25T14:02:13.192664+0000","last_unstale":"2024-04-25T14:02:14.620239+0000","last_undegraded":"2024-04-25T14:02:14.620239+0000","last_fullsized":"2024-04-25T14:02:14.620239+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621118+0000","last_change":"2024-04-25T14:02:09.175890+0000","last_active":"2024-04-25T14:02:14.621118+0000","last_peered":"2024-04-25T14:02:14.621118+0000","last_clean":"2024-04-25T14:02:14.621118+0000","last_became_active":"2024-04-25T14:02:09.175586+0000","last_became_peered":"2024-04-25T14:02:09.175586+0000","last_unstale":"2024-04-25T14:02:14.621118+0000","last_undegraded":"2024-04-25T14:02:14.621118+0000","last_fullsized":"2024-04-25T14:02:14.621118+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619271+0000","last_change":"2024-04-25T14:02:08.141597+0000","last_active":"2024-04-25T14:02:14.619271+0000","last_peered":"2024-04-25T14:02:14.619271+0000","last_clean":"2024-04-25T14:02:14.619271+0000","last_became_active":"2024-04-25T14:02:08.141271+0000","last_became_peered":"2024-04-25T14:02:08.141271+0000","last_unstale":"2024-04-25T14:02:14.619271+0000","last_undegraded":"2024-04-25T14:02:14.619271+0000","last_fullsized":"2024-04-25T14:02:14.619271+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619661+0000","last_change":"2024-04-25T14:02:13.208359+0000","last_active":"2024-04-25T14:02:14.619661+0000","last_peered":"2024-04-25T14:02:14.619661+0000","last_clean":"2024-04-25T14:02:14.619661+0000","last_became_active":"2024-04-25T14:02:13.204049+0000","last_became_peered":"2024-04-25T14:02:13.204049+0000","last_unstale":"2024-04-25T14:02:14.619661+0000","last_undegraded":"2024-04-25T14:02:14.619661+0000","last_fullsized":"2024-04-25T14:02:14.619661+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620199+0000","last_change":"2024-04-25T14:02:09.153508+0000","last_active":"2024-04-25T14:02:14.620199+0000","last_peered":"2024-04-25T14:02:14.620199+0000","last_clean":"2024-04-25T14:02:14.620199+0000","last_became_active":"2024-04-25T14:02:09.153087+0000","last_became_peered":"2024-04-25T14:02:09.153087+0000","last_unstale":"2024-04-25T14:02:14.620199+0000","last_undegraded":"2024-04-25T14:02:14.620199+0000","last_fullsized":"2024-04-25T14:02:14.620199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181562+0000","last_change":"2024-04-25T14:02:08.149363+0000","last_active":"2024-04-25T14:02:14.181562+0000","last_peered":"2024-04-25T14:02:14.181562+0000","last_clean":"2024-04-25T14:02:14.181562+0000","last_became_active":"2024-04-25T14:02:08.148936+0000","last_became_peered":"2024-04-25T14:02:08.148936+0000","last_unstale":"2024-04-25T14:02:14.181562+0000","last_undegraded":"2024-04-25T14:02:14.181562+0000","last_fullsized":"2024-04-25T14:02:14.181562+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:47:33.642884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180089+0000","last_change":"2024-04-25T14:02:13.201594+0000","last_active":"2024-04-25T14:02:14.180089+0000","last_peered":"2024-04-25T14:02:14.180089+0000","last_clean":"2024-04-25T14:02:14.180089+0000","last_became_active":"2024-04-25T14:02:13.201156+0000","last_became_peered":"2024-04-25T14:02:13.201156+0000","last_unstale":"2024-04-25T14:02:14.180089+0000","last_undegraded":"2024-04-25T14:02:14.180089+0000","last_fullsized":"2024-04-25T14:02:14.180089+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620504+0000","last_change":"2024-04-25T14:02:09.164648+0000","last_active":"2024-04-25T14:02:14.620504+0000","last_peered":"2024-04-25T14:02:14.620504+0000","last_clean":"2024-04-25T14:02:14.620504+0000","last_became_active":"2024-04-25T14:02:09.164319+0000","last_became_peered":"2024-04-25T14:02:09.164319+0000","last_unstale":"2024-04-25T14:02:14.620504+0000","last_undegraded":"2024-04-25T14:02:14.620504+0000","last_fullsized":"2024-04-25T14:02:14.620504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620863+0000","last_change":"2024-04-25T14:02:08.148046+0000","last_active":"2024-04-25T14:02:14.620863+0000","last_peered":"2024-04-25T14:02:14.620863+0000","last_clean":"2024-04-25T14:02:14.620863+0000","last_became_active":"2024-04-25T14:02:08.147707+0000","last_became_peered":"2024-04-25T14:02:08.147707+0000","last_unstale":"2024-04-25T14:02:14.620863+0000","last_undegraded":"2024-04-25T14:02:14.620863+0000","last_fullsized":"2024-04-25T14:02:14.620863+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619751+0000","last_change":"2024-04-25T14:02:13.197572+0000","last_active":"2024-04-25T14:02:14.619751+0000","last_peered":"2024-04-25T14:02:14.619751+0000","last_clean":"2024-04-25T14:02:14.619751+0000","last_became_active":"2024-04-25T14:02:13.197138+0000","last_became_peered":"2024-04-25T14:02:13.197138+0000","last_unstale":"2024-04-25T14:02:14.619751+0000","last_undegraded":"2024-04-25T14:02:14.619751+0000","last_fullsized":"2024-04-25T14:02:14.619751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619861+0000","last_change":"2024-04-25T14:02:09.163807+0000","last_active":"2024-04-25T14:02:14.619861+0000","last_peered":"2024-04-25T14:02:14.619861+0000","last_clean":"2024-04-25T14:02:14.619861+0000","last_became_active":"2024-04-25T14:02:09.163510+0000","last_became_peered":"2024-04-25T14:02:09.163510+0000","last_unstale":"2024-04-25T14:02:14.619861+0000","last_undegraded":"2024-04-25T14:02:14.619861+0000","last_fullsized":"2024-04-25T14:02:14.619861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620015+0000","last_change":"2024-04-25T14:02:08.148684+0000","last_active":"2024-04-25T14:02:14.620015+0000","last_peered":"2024-04-25T14:02:14.620015+0000","last_clean":"2024-04-25T14:02:14.620015+0000","last_became_active":"2024-04-25T14:02:08.148384+0000","last_became_peered":"2024-04-25T14:02:08.148384+0000","last_unstale":"2024-04-25T14:02:14.620015+0000","last_undegraded":"2024-04-25T14:02:14.620015+0000","last_fullsized":"2024-04-25T14:02:14.620015+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620837+0000","last_change":"2024-04-25T14:02:13.204849+0000","last_active":"2024-04-25T14:02:14.620837+0000","last_peered":"2024-04-25T14:02:14.620837+0000","last_clean":"2024-04-25T14:02:14.620837+0000","last_became_active":"2024-04-25T14:02:13.204459+0000","last_became_peered":"2024-04-25T14:02:13.204459+0000","last_unstale":"2024-04-25T14:02:14.620837+0000","last_undegraded":"2024-04-25T14:02:14.620837+0000","last_fullsized":"2024-04-25T14:02:14.620837+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619156+0000","last_change":"2024-04-25T14:02:09.153461+0000","last_active":"2024-04-25T14:02:14.619156+0000","last_peered":"2024-04-25T14:02:14.619156+0000","last_clean":"2024-04-25T14:02:14.619156+0000","last_became_active":"2024-04-25T14:02:09.153122+0000","last_became_peered":"2024-04-25T14:02:09.153122+0000","last_unstale":"2024-04-25T14:02:14.619156+0000","last_undegraded":"2024-04-25T14:02:14.619156+0000","last_fullsized":"2024-04-25T14:02:14.619156+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619253+0000","last_change":"2024-04-25T14:02:08.141488+0000","last_active":"2024-04-25T14:02:14.619253+0000","last_peered":"2024-04-25T14:02:14.619253+0000","last_clean":"2024-04-25T14:02:14.619253+0000","last_became_active":"2024-04-25T14:02:08.141195+0000","last_became_peered":"2024-04-25T14:02:08.141195+0000","last_unstale":"2024-04-25T14:02:14.619253+0000","last_undegraded":"2024-04-25T14:02:14.619253+0000","last_fullsized":"2024-04-25T14:02:14.619253+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620173+0000","last_change":"2024-04-25T14:02:13.202926+0000","last_active":"2024-04-25T14:02:14.620173+0000","last_peered":"2024-04-25T14:02:14.620173+0000","last_clean":"2024-04-25T14:02:14.620173+0000","last_became_active":"2024-04-25T14:02:13.202452+0000","last_became_peered":"2024-04-25T14:02:13.202452+0000","last_unstale":"2024-04-25T14:02:14.620173+0000","last_undegraded":"2024-04-25T14:02:14.620173+0000","last_fullsized":"2024-04-25T14:02:14.620173+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179693+0000","last_change":"2024-04-25T14:02:09.162133+0000","last_active":"2024-04-25T14:02:14.179693+0000","last_peered":"2024-04-25T14:02:14.179693+0000","last_clean":"2024-04-25T14:02:14.179693+0000","last_became_active":"2024-04-25T14:02:09.161795+0000","last_became_peered":"2024-04-25T14:02:09.161795+0000","last_unstale":"2024-04-25T14:02:14.179693+0000","last_undegraded":"2024-04-25T14:02:14.179693+0000","last_fullsized":"2024-04-25T14:02:14.179693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:49.968260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180083+0000","last_change":"2024-04-25T14:02:08.144670+0000","last_active":"2024-04-25T14:02:14.180083+0000","last_peered":"2024-04-25T14:02:14.180083+0000","last_clean":"2024-04-25T14:02:14.180083+0000","last_became_active":"2024-04-25T14:02:08.143303+0000","last_became_peered":"2024-04-25T14:02:08.143303+0000","last_unstale":"2024-04-25T14:02:14.180083+0000","last_undegraded":"2024-04-25T14:02:14.180083+0000","last_fullsized":"2024-04-25T14:02:14.180083+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:37.301845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.199069+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.198521+0000","last_became_peered":"2024-04-25T14:02:13.198521+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:25.385124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.164561+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.164215+0000","last_became_peered":"2024-04-25T14:02:09.164215+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619618+0000","last_change":"2024-04-25T14:02:08.150157+0000","last_active":"2024-04-25T14:02:14.619618+0000","last_peered":"2024-04-25T14:02:14.619618+0000","last_clean":"2024-04-25T14:02:14.619618+0000","last_became_active":"2024-04-25T14:02:08.149812+0000","last_became_peered":"2024-04-25T14:02:08.149812+0000","last_unstale":"2024-04-25T14:02:14.619618+0000","last_undegraded":"2024-04-25T14:02:14.619618+0000","last_fullsized":"2024-04-25T14:02:14.619618+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":118,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180256+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:02:14.180256+0000","last_peered":"2024-04-25T14:02:14.180256+0000","last_clean":"2024-04-25T14:02:14.180256+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:02:14.180256+0000","last_undegraded":"2024-04-25T14:02:14.180256+0000","last_fullsized":"2024-04-25T14:02:14.180256+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180746+0000","last_change":"2024-04-25T14:02:13.202985+0000","last_active":"2024-04-25T14:02:14.180746+0000","last_peered":"2024-04-25T14:02:14.180746+0000","last_clean":"2024-04-25T14:02:14.180746+0000","last_became_active":"2024-04-25T14:02:13.202514+0000","last_became_peered":"2024-04-25T14:02:13.202514+0000","last_unstale":"2024-04-25T14:02:14.180746+0000","last_undegraded":"2024-04-25T14:02:14.180746+0000","last_fullsized":"2024-04-25T14:02:14.180746+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:39.129451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181284+0000","last_change":"2024-04-25T14:02:09.164697+0000","last_active":"2024-04-25T14:02:14.181284+0000","last_peered":"2024-04-25T14:02:14.181284+0000","last_clean":"2024-04-25T14:02:14.181284+0000","last_became_active":"2024-04-25T14:02:09.164352+0000","last_became_peered":"2024-04-25T14:02:09.164352+0000","last_unstale":"2024-04-25T14:02:14.181284+0000","last_undegraded":"2024-04-25T14:02:14.181284+0000","last_fullsized":"2024-04-25T14:02:14.181284+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:35.083949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620427+0000","last_change":"2024-04-25T14:02:08.150437+0000","last_active":"2024-04-25T14:02:14.620427+0000","last_peered":"2024-04-25T14:02:14.620427+0000","last_clean":"2024-04-25T14:02:14.620427+0000","last_became_active":"2024-04-25T14:02:08.149954+0000","last_became_peered":"2024-04-25T14:02:08.149954+0000","last_unstale":"2024-04-25T14:02:14.620427+0000","last_undegraded":"2024-04-25T14:02:14.620427+0000","last_fullsized":"2024-04-25T14:02:14.620427+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181863+0000","last_change":"2024-04-25T14:02:13.198918+0000","last_active":"2024-04-25T14:02:14.181863+0000","last_peered":"2024-04-25T14:02:14.181863+0000","last_clean":"2024-04-25T14:02:14.181863+0000","last_became_active":"2024-04-25T14:02:13.198244+0000","last_became_peered":"2024-04-25T14:02:13.198244+0000","last_unstale":"2024-04-25T14:02:14.181863+0000","last_undegraded":"2024-04-25T14:02:14.181863+0000","last_fullsized":"2024-04-25T14:02:14.181863+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619706+0000","last_change":"2024-04-25T14:02:09.162899+0000","last_active":"2024-04-25T14:02:14.619706+0000","last_peered":"2024-04-25T14:02:14.619706+0000","last_clean":"2024-04-25T14:02:14.619706+0000","last_became_active":"2024-04-25T14:02:09.162573+0000","last_became_peered":"2024-04-25T14:02:09.162573+0000","last_unstale":"2024-04-25T14:02:14.619706+0000","last_undegraded":"2024-04-25T14:02:14.619706+0000","last_fullsized":"2024-04-25T14:02:14.619706+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621027+0000","last_change":"2024-04-25T14:02:08.140157+0000","last_active":"2024-04-25T14:02:14.621027+0000","last_peered":"2024-04-25T14:02:14.621027+0000","last_clean":"2024-04-25T14:02:14.621027+0000","last_became_active":"2024-04-25T14:02:08.139477+0000","last_became_peered":"2024-04-25T14:02:08.139477+0000","last_unstale":"2024-04-25T14:02:14.621027+0000","last_undegraded":"2024-04-25T14:02:14.621027+0000","last_fullsized":"2024-04-25T14:02:14.621027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:38.500335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180180+0000","last_change":"2024-04-25T14:02:13.201803+0000","last_active":"2024-04-25T14:02:14.180180+0000","last_peered":"2024-04-25T14:02:14.180180+0000","last_clean":"2024-04-25T14:02:14.180180+0000","last_became_active":"2024-04-25T14:02:13.201388+0000","last_became_peered":"2024-04-25T14:02:13.201388+0000","last_unstale":"2024-04-25T14:02:14.180180+0000","last_undegraded":"2024-04-25T14:02:14.180180+0000","last_fullsized":"2024-04-25T14:02:14.180180+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:39.249360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182344+0000","last_change":"2024-04-25T14:02:09.171757+0000","last_active":"2024-04-25T14:02:14.182344+0000","last_peered":"2024-04-25T14:02:14.182344+0000","last_clean":"2024-04-25T14:02:14.182344+0000","last_became_active":"2024-04-25T14:02:09.168908+0000","last_became_peered":"2024-04-25T14:02:09.168908+0000","last_unstale":"2024-04-25T14:02:14.182344+0000","last_undegraded":"2024-04-25T14:02:14.182344+0000","last_fullsized":"2024-04-25T14:02:14.182344+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:45:21.339622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":24,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620710+0000","last_change":"2024-04-25T14:02:08.140886+0000","last_active":"2024-04-25T14:02:14.620710+0000","last_peered":"2024-04-25T14:02:14.620710+0000","last_clean":"2024-04-25T14:02:14.620710+0000","last_became_active":"2024-04-25T14:02:08.138585+0000","last_became_peered":"2024-04-25T14:02:08.138585+0000","last_unstale":"2024-04-25T14:02:14.620710+0000","last_undegraded":"2024-04-25T14:02:14.620710+0000","last_fullsized":"2024-04-25T14:02:14.620710+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619005+0000","last_change":"2024-04-25T14:02:13.194033+0000","last_active":"2024-04-25T14:02:14.619005+0000","last_peered":"2024-04-25T14:02:14.619005+0000","last_clean":"2024-04-25T14:02:14.619005+0000","last_became_active":"2024-04-25T14:02:13.193486+0000","last_became_peered":"2024-04-25T14:02:13.193486+0000","last_unstale":"2024-04-25T14:02:14.619005+0000","last_undegraded":"2024-04-25T14:02:14.619005+0000","last_fullsized":"2024-04-25T14:02:14.619005+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:57:08.249655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181242+0000","last_change":"2024-04-25T14:02:09.154176+0000","last_active":"2024-04-25T14:02:14.181242+0000","last_peered":"2024-04-25T14:02:14.181242+0000","last_clean":"2024-04-25T14:02:14.181242+0000","last_became_active":"2024-04-25T14:02:09.153867+0000","last_became_peered":"2024-04-25T14:02:09.153867+0000","last_unstale":"2024-04-25T14:02:14.181242+0000","last_undegraded":"2024-04-25T14:02:14.181242+0000","last_fullsized":"2024-04-25T14:02:14.181242+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619639+0000","last_change":"2024-04-25T14:02:08.148162+0000","last_active":"2024-04-25T14:02:14.619639+0000","last_peered":"2024-04-25T14:02:14.619639+0000","last_clean":"2024-04-25T14:02:14.619639+0000","last_became_active":"2024-04-25T14:02:08.147822+0000","last_became_peered":"2024-04-25T14:02:08.147822+0000","last_unstale":"2024-04-25T14:02:14.619639+0000","last_undegraded":"2024-04-25T14:02:14.619639+0000","last_fullsized":"2024-04-25T14:02:14.619639+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:04:47.001110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619644+0000","last_change":"2024-04-25T14:02:13.204464+0000","last_active":"2024-04-25T14:02:14.619644+0000","last_peered":"2024-04-25T14:02:14.619644+0000","last_clean":"2024-04-25T14:02:14.619644+0000","last_became_active":"2024-04-25T14:02:13.203501+0000","last_became_peered":"2024-04-25T14:02:13.203501+0000","last_unstale":"2024-04-25T14:02:14.619644+0000","last_undegraded":"2024-04-25T14:02:14.619644+0000","last_fullsized":"2024-04-25T14:02:14.619644+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:36.450612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180471+0000","last_change":"2024-04-25T14:02:09.169427+0000","last_active":"2024-04-25T14:02:14.180471+0000","last_peered":"2024-04-25T14:02:14.180471+0000","last_clean":"2024-04-25T14:02:14.180471+0000","last_became_active":"2024-04-25T14:02:09.168920+0000","last_became_peered":"2024-04-25T14:02:09.168920+0000","last_unstale":"2024-04-25T14:02:14.180471+0000","last_undegraded":"2024-04-25T14:02:14.180471+0000","last_fullsized":"2024-04-25T14:02:14.180471+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:53:08.333998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618690+0000","last_change":"2024-04-25T14:02:08.143121+0000","last_active":"2024-04-25T14:02:14.618690+0000","last_peered":"2024-04-25T14:02:14.618690+0000","last_clean":"2024-04-25T14:02:14.618690+0000","last_became_active":"2024-04-25T14:02:08.142765+0000","last_became_peered":"2024-04-25T14:02:08.142765+0000","last_unstale":"2024-04-25T14:02:14.618690+0000","last_undegraded":"2024-04-25T14:02:14.618690+0000","last_fullsized":"2024-04-25T14:02:14.618690+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180517+0000","last_change":"2024-04-25T14:02:13.205101+0000","last_active":"2024-04-25T14:02:14.180517+0000","last_peered":"2024-04-25T14:02:14.180517+0000","last_clean":"2024-04-25T14:02:14.180517+0000","last_became_active":"2024-04-25T14:02:13.204696+0000","last_became_peered":"2024-04-25T14:02:13.204696+0000","last_unstale":"2024-04-25T14:02:14.180517+0000","last_undegraded":"2024-04-25T14:02:14.180517+0000","last_fullsized":"2024-04-25T14:02:14.180517+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:38.959916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179895+0000","last_change":"2024-04-25T14:02:09.165439+0000","last_active":"2024-04-25T14:02:14.179895+0000","last_peered":"2024-04-25T14:02:14.179895+0000","last_clean":"2024-04-25T14:02:14.179895+0000","last_became_active":"2024-04-25T14:02:09.164934+0000","last_became_peered":"2024-04-25T14:02:09.164934+0000","last_unstale":"2024-04-25T14:02:14.179895+0000","last_undegraded":"2024-04-25T14:02:14.179895+0000","last_fullsized":"2024-04-25T14:02:14.179895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180810+0000","last_change":"2024-04-25T14:02:08.144013+0000","last_active":"2024-04-25T14:02:14.180810+0000","last_peered":"2024-04-25T14:02:14.180810+0000","last_clean":"2024-04-25T14:02:14.180810+0000","last_became_active":"2024-04-25T14:02:08.141723+0000","last_became_peered":"2024-04-25T14:02:08.141723+0000","last_unstale":"2024-04-25T14:02:14.180810+0000","last_undegraded":"2024-04-25T14:02:14.180810+0000","last_fullsized":"2024-04-25T14:02:14.180810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620974+0000","last_change":"2024-04-25T14:02:13.199511+0000","last_active":"2024-04-25T14:02:14.620974+0000","last_peered":"2024-04-25T14:02:14.620974+0000","last_clean":"2024-04-25T14:02:14.620974+0000","last_became_active":"2024-04-25T14:02:13.199074+0000","last_became_peered":"2024-04-25T14:02:13.199074+0000","last_unstale":"2024-04-25T14:02:14.620974+0000","last_undegraded":"2024-04-25T14:02:14.620974+0000","last_fullsized":"2024-04-25T14:02:14.620974+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:25:47.660769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180811+0000","last_change":"2024-04-25T14:02:09.171097+0000","last_active":"2024-04-25T14:02:14.180811+0000","last_peered":"2024-04-25T14:02:14.180811+0000","last_clean":"2024-04-25T14:02:14.180811+0000","last_became_active":"2024-04-25T14:02:09.169656+0000","last_became_peered":"2024-04-25T14:02:09.169656+0000","last_unstale":"2024-04-25T14:02:14.180811+0000","last_undegraded":"2024-04-25T14:02:14.180811+0000","last_fullsized":"2024-04-25T14:02:14.180811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:46:34.914414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619583+0000","last_change":"2024-04-25T14:02:08.148427+0000","last_active":"2024-04-25T14:02:14.619583+0000","last_peered":"2024-04-25T14:02:14.619583+0000","last_clean":"2024-04-25T14:02:14.619583+0000","last_became_active":"2024-04-25T14:02:08.148019+0000","last_became_peered":"2024-04-25T14:02:08.148019+0000","last_unstale":"2024-04-25T14:02:14.619583+0000","last_undegraded":"2024-04-25T14:02:14.619583+0000","last_fullsized":"2024-04-25T14:02:14.619583+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621238+0000","last_change":"2024-04-25T14:02:13.203865+0000","last_active":"2024-04-25T14:02:14.621238+0000","last_peered":"2024-04-25T14:02:14.621238+0000","last_clean":"2024-04-25T14:02:14.621238+0000","last_became_active":"2024-04-25T14:02:13.203056+0000","last_became_peered":"2024-04-25T14:02:13.203056+0000","last_unstale":"2024-04-25T14:02:14.621238+0000","last_undegraded":"2024-04-25T14:02:14.621238+0000","last_fullsized":"2024-04-25T14:02:14.621238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:17:58.555213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179953+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.179953+0000","last_peered":"2024-04-25T14:02:14.179953+0000","last_clean":"2024-04-25T14:02:14.179953+0000","last_became_active":"2024-04-25T14:02:09.171025+0000","last_became_peered":"2024-04-25T14:02:09.171025+0000","last_unstale":"2024-04-25T14:02:14.179953+0000","last_undegraded":"2024-04-25T14:02:14.179953+0000","last_fullsized":"2024-04-25T14:02:14.179953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180403+0000","last_change":"2024-04-25T14:02:08.141709+0000","last_active":"2024-04-25T14:02:14.180403+0000","last_peered":"2024-04-25T14:02:14.180403+0000","last_clean":"2024-04-25T14:02:14.180403+0000","last_became_active":"2024-04-25T14:02:08.141389+0000","last_became_peered":"2024-04-25T14:02:08.141389+0000","last_unstale":"2024-04-25T14:02:14.180403+0000","last_undegraded":"2024-04-25T14:02:14.180403+0000","last_fullsized":"2024-04-25T14:02:14.180403+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:52:53.806197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620299+0000","last_change":"2024-04-25T14:02:13.205138+0000","last_active":"2024-04-25T14:02:14.620299+0000","last_peered":"2024-04-25T14:02:14.620299+0000","last_clean":"2024-04-25T14:02:14.620299+0000","last_became_active":"2024-04-25T14:02:13.204712+0000","last_became_peered":"2024-04-25T14:02:13.204712+0000","last_unstale":"2024-04-25T14:02:14.620299+0000","last_undegraded":"2024-04-25T14:02:14.620299+0000","last_fullsized":"2024-04-25T14:02:14.620299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181162+0000","last_change":"2024-04-25T14:02:09.159739+0000","last_active":"2024-04-25T14:02:14.181162+0000","last_peered":"2024-04-25T14:02:14.181162+0000","last_clean":"2024-04-25T14:02:14.181162+0000","last_became_active":"2024-04-25T14:02:09.158156+0000","last_became_peered":"2024-04-25T14:02:09.158156+0000","last_unstale":"2024-04-25T14:02:14.181162+0000","last_undegraded":"2024-04-25T14:02:14.181162+0000","last_fullsized":"2024-04-25T14:02:14.181162+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:30.748746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"24'9","reported_seq":30,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:18.690850+0000","last_change":"2024-04-25T14:02:08.143908+0000","last_active":"2024-04-25T14:02:18.690850+0000","last_peered":"2024-04-25T14:02:18.690850+0000","last_clean":"2024-04-25T14:02:18.690850+0000","last_became_active":"2024-04-25T14:02:08.143478+0000","last_became_peered":"2024-04-25T14:02:08.143478+0000","last_unstale":"2024-04-25T14:02:18.690850+0000","last_undegraded":"2024-04-25T14:02:18.690850+0000","last_fullsized":"2024-04-25T14:02:18.690850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619926+0000","last_change":"2024-04-25T14:02:13.197367+0000","last_active":"2024-04-25T14:02:14.619926+0000","last_peered":"2024-04-25T14:02:14.619926+0000","last_clean":"2024-04-25T14:02:14.619926+0000","last_became_active":"2024-04-25T14:02:13.197029+0000","last_became_peered":"2024-04-25T14:02:13.197029+0000","last_unstale":"2024-04-25T14:02:14.619926+0000","last_undegraded":"2024-04-25T14:02:14.619926+0000","last_fullsized":"2024-04-25T14:02:14.619926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:52.267144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620116+0000","last_change":"2024-04-25T14:02:09.162706+0000","last_active":"2024-04-25T14:02:14.620116+0000","last_peered":"2024-04-25T14:02:14.620116+0000","last_clean":"2024-04-25T14:02:14.620116+0000","last_became_active":"2024-04-25T14:02:09.162316+0000","last_became_peered":"2024-04-25T14:02:09.162316+0000","last_unstale":"2024-04-25T14:02:14.620116+0000","last_undegraded":"2024-04-25T14:02:14.620116+0000","last_fullsized":"2024-04-25T14:02:14.620116+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:43:23.993603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":25,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620975+0000","last_change":"2024-04-25T14:02:08.150406+0000","last_active":"2024-04-25T14:02:14.620975+0000","last_peered":"2024-04-25T14:02:14.620975+0000","last_clean":"2024-04-25T14:02:14.620975+0000","last_became_active":"2024-04-25T14:02:08.150057+0000","last_became_peered":"2024-04-25T14:02:08.150057+0000","last_unstale":"2024-04-25T14:02:14.620975+0000","last_undegraded":"2024-04-25T14:02:14.620975+0000","last_fullsized":"2024-04-25T14:02:14.620975+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:20.067841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619602+0000","last_change":"2024-04-25T14:02:13.197851+0000","last_active":"2024-04-25T14:02:14.619602+0000","last_peered":"2024-04-25T14:02:14.619602+0000","last_clean":"2024-04-25T14:02:14.619602+0000","last_became_active":"2024-04-25T14:02:13.197109+0000","last_became_peered":"2024-04-25T14:02:13.197109+0000","last_unstale":"2024-04-25T14:02:14.619602+0000","last_undegraded":"2024-04-25T14:02:14.619602+0000","last_fullsized":"2024-04-25T14:02:14.619602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:20.798044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180425+0000","last_change":"2024-04-25T14:02:09.168810+0000","last_active":"2024-04-25T14:02:14.180425+0000","last_peered":"2024-04-25T14:02:14.180425+0000","last_clean":"2024-04-25T14:02:14.180425+0000","last_became_active":"2024-04-25T14:02:09.168323+0000","last_became_peered":"2024-04-25T14:02:09.168323+0000","last_unstale":"2024-04-25T14:02:14.180425+0000","last_undegraded":"2024-04-25T14:02:14.180425+0000","last_fullsized":"2024-04-25T14:02:14.180425+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619958+0000","last_change":"2024-04-25T14:02:08.145115+0000","last_active":"2024-04-25T14:02:14.619958+0000","last_peered":"2024-04-25T14:02:14.619958+0000","last_clean":"2024-04-25T14:02:14.619958+0000","last_became_active":"2024-04-25T14:02:08.144853+0000","last_became_peered":"2024-04-25T14:02:08.144853+0000","last_unstale":"2024-04-25T14:02:14.619958+0000","last_undegraded":"2024-04-25T14:02:14.619958+0000","last_fullsized":"2024-04-25T14:02:14.619958+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:31:30.387482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619831+0000","last_change":"2024-04-25T14:02:13.203422+0000","last_active":"2024-04-25T14:02:14.619831+0000","last_peered":"2024-04-25T14:02:14.619831+0000","last_clean":"2024-04-25T14:02:14.619831+0000","last_became_active":"2024-04-25T14:02:13.203003+0000","last_became_peered":"2024-04-25T14:02:13.203003+0000","last_unstale":"2024-04-25T14:02:14.619831+0000","last_undegraded":"2024-04-25T14:02:14.619831+0000","last_fullsized":"2024-04-25T14:02:14.619831+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:02:28.366813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619921+0000","last_change":"2024-04-25T14:02:09.171494+0000","last_active":"2024-04-25T14:02:14.619921+0000","last_peered":"2024-04-25T14:02:14.619921+0000","last_clean":"2024-04-25T14:02:14.619921+0000","last_became_active":"2024-04-25T14:02:09.171163+0000","last_became_peered":"2024-04-25T14:02:09.171163+0000","last_unstale":"2024-04-25T14:02:14.619921+0000","last_undegraded":"2024-04-25T14:02:14.619921+0000","last_fullsized":"2024-04-25T14:02:14.619921+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181248+0000","last_change":"2024-04-25T14:02:08.150536+0000","last_active":"2024-04-25T14:02:14.181248+0000","last_peered":"2024-04-25T14:02:14.181248+0000","last_clean":"2024-04-25T14:02:14.181248+0000","last_became_active":"2024-04-25T14:02:08.150156+0000","last_became_peered":"2024-04-25T14:02:08.150156+0000","last_unstale":"2024-04-25T14:02:14.181248+0000","last_undegraded":"2024-04-25T14:02:14.181248+0000","last_fullsized":"2024-04-25T14:02:14.181248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:19:10.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619292+0000","last_change":"2024-04-25T14:02:13.201000+0000","last_active":"2024-04-25T14:02:14.619292+0000","last_peered":"2024-04-25T14:02:14.619292+0000","last_clean":"2024-04-25T14:02:14.619292+0000","last_became_active":"2024-04-25T14:02:13.200594+0000","last_became_peered":"2024-04-25T14:02:13.200594+0000","last_unstale":"2024-04-25T14:02:14.619292+0000","last_undegraded":"2024-04-25T14:02:14.619292+0000","last_fullsized":"2024-04-25T14:02:14.619292+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:20:34.706373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180405+0000","last_change":"2024-04-25T14:02:13.204286+0000","last_active":"2024-04-25T14:02:14.180405+0000","last_peered":"2024-04-25T14:02:14.180405+0000","last_clean":"2024-04-25T14:02:14.180405+0000","last_became_active":"2024-04-25T14:02:13.203885+0000","last_became_peered":"2024-04-25T14:02:13.203885+0000","last_unstale":"2024-04-25T14:02:14.180405+0000","last_undegraded":"2024-04-25T14:02:14.180405+0000","last_fullsized":"2024-04-25T14:02:14.180405+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:11.432357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619363+0000","last_change":"2024-04-25T14:02:09.158647+0000","last_active":"2024-04-25T14:02:14.619363+0000","last_peered":"2024-04-25T14:02:14.619363+0000","last_clean":"2024-04-25T14:02:14.619363+0000","last_became_active":"2024-04-25T14:02:09.158307+0000","last_became_peered":"2024-04-25T14:02:09.158307+0000","last_unstale":"2024-04-25T14:02:14.619363+0000","last_undegraded":"2024-04-25T14:02:14.619363+0000","last_fullsized":"2024-04-25T14:02:14.619363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619239+0000","last_change":"2024-04-25T14:02:08.149002+0000","last_active":"2024-04-25T14:02:14.619239+0000","last_peered":"2024-04-25T14:02:14.619239+0000","last_clean":"2024-04-25T14:02:14.619239+0000","last_became_active":"2024-04-25T14:02:08.148700+0000","last_became_peered":"2024-04-25T14:02:08.148700+0000","last_unstale":"2024-04-25T14:02:14.619239+0000","last_undegraded":"2024-04-25T14:02:14.619239+0000","last_fullsized":"2024-04-25T14:02:14.619239+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:12:36.403204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181096+0000","last_change":"2024-04-25T14:02:13.197649+0000","last_active":"2024-04-25T14:02:14.181096+0000","last_peered":"2024-04-25T14:02:14.181096+0000","last_clean":"2024-04-25T14:02:14.181096+0000","last_became_active":"2024-04-25T14:02:13.197203+0000","last_became_peered":"2024-04-25T14:02:13.197203+0000","last_unstale":"2024-04-25T14:02:14.181096+0000","last_undegraded":"2024-04-25T14:02:14.181096+0000","last_fullsized":"2024-04-25T14:02:14.181096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:18.214121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619786+0000","last_change":"2024-04-25T14:02:09.164726+0000","last_active":"2024-04-25T14:02:14.619786+0000","last_peered":"2024-04-25T14:02:14.619786+0000","last_clean":"2024-04-25T14:02:14.619786+0000","last_became_active":"2024-04-25T14:02:09.164047+0000","last_became_peered":"2024-04-25T14:02:09.164047+0000","last_unstale":"2024-04-25T14:02:14.619786+0000","last_undegraded":"2024-04-25T14:02:14.619786+0000","last_fullsized":"2024-04-25T14:02:14.619786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619586+0000","last_change":"2024-04-25T14:02:08.138596+0000","last_active":"2024-04-25T14:02:14.619586+0000","last_peered":"2024-04-25T14:02:14.619586+0000","last_clean":"2024-04-25T14:02:14.619586+0000","last_became_active":"2024-04-25T14:02:08.138268+0000","last_became_peered":"2024-04-25T14:02:08.138268+0000","last_unstale":"2024-04-25T14:02:14.619586+0000","last_undegraded":"2024-04-25T14:02:14.619586+0000","last_fullsized":"2024-04-25T14:02:14.619586+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182061+0000","last_change":"2024-04-25T14:02:08.144180+0000","last_active":"2024-04-25T14:02:14.182061+0000","last_peered":"2024-04-25T14:02:14.182061+0000","last_clean":"2024-04-25T14:02:14.182061+0000","last_became_active":"2024-04-25T14:02:08.142554+0000","last_became_peered":"2024-04-25T14:02:08.142554+0000","last_unstale":"2024-04-25T14:02:14.182061+0000","last_undegraded":"2024-04-25T14:02:14.182061+0000","last_fullsized":"2024-04-25T14:02:14.182061+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:53.249209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.163851+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.163561+0000","last_became_peered":"2024-04-25T14:02:09.163561+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620647+0000","last_change":"2024-04-25T14:02:13.200670+0000","last_active":"2024-04-25T14:02:14.620647+0000","last_peered":"2024-04-25T14:02:14.620647+0000","last_clean":"2024-04-25T14:02:14.620647+0000","last_became_active":"2024-04-25T14:02:13.200255+0000","last_became_peered":"2024-04-25T14:02:13.200255+0000","last_unstale":"2024-04-25T14:02:14.620647+0000","last_undegraded":"2024-04-25T14:02:14.620647+0000","last_fullsized":"2024-04-25T14:02:14.620647+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:25:49.451815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":23,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620669+0000","last_change":"2024-04-25T14:02:08.136214+0000","last_active":"2024-04-25T14:02:14.620669+0000","last_peered":"2024-04-25T14:02:14.620669+0000","last_clean":"2024-04-25T14:02:14.620669+0000","last_became_active":"2024-04-25T14:02:08.135892+0000","last_became_peered":"2024-04-25T14:02:08.135892+0000","last_unstale":"2024-04-25T14:02:14.620669+0000","last_undegraded":"2024-04-25T14:02:14.620669+0000","last_fullsized":"2024-04-25T14:02:14.620669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:07.113445+0000","last_clean_scrub_stamp":"2024-04-25T14:02:07.113445+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:00.331661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620404+0000","last_change":"2024-04-25T14:02:09.160067+0000","last_active":"2024-04-25T14:02:14.620404+0000","last_peered":"2024-04-25T14:02:14.620404+0000","last_clean":"2024-04-25T14:02:14.620404+0000","last_became_active":"2024-04-25T14:02:09.159705+0000","last_became_peered":"2024-04-25T14:02:09.159705+0000","last_unstale":"2024-04-25T14:02:14.620404+0000","last_undegraded":"2024-04-25T14:02:14.620404+0000","last_fullsized":"2024-04-25T14:02:14.620404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:07:21.073575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621148+0000","last_change":"2024-04-25T14:02:13.203327+0000","last_active":"2024-04-25T14:02:14.621148+0000","last_peered":"2024-04-25T14:02:14.621148+0000","last_clean":"2024-04-25T14:02:14.621148+0000","last_became_active":"2024-04-25T14:02:13.202870+0000","last_became_peered":"2024-04-25T14:02:13.202870+0000","last_unstale":"2024-04-25T14:02:14.621148+0000","last_undegraded":"2024-04-25T14:02:14.621148+0000","last_fullsized":"2024-04-25T14:02:14.621148+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:12.164680+0000","last_clean_scrub_stamp":"2024-04-25T14:02:12.164680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:06:39.636544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179770+0000","last_change":"2024-04-25T14:02:09.165255+0000","last_active":"2024-04-25T14:02:14.179770+0000","last_peered":"2024-04-25T14:02:14.179770+0000","last_clean":"2024-04-25T14:02:14.179770+0000","last_became_active":"2024-04-25T14:02:09.164886+0000","last_became_peered":"2024-04-25T14:02:09.164886+0000","last_unstale":"2024-04-25T14:02:14.179770+0000","last_undegraded":"2024-04-25T14:02:14.179770+0000","last_fullsized":"2024-04-25T14:02:14.179770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619161+0000","last_change":"2024-04-25T14:02:09.161880+0000","last_active":"2024-04-25T14:02:14.619161+0000","last_peered":"2024-04-25T14:02:14.619161+0000","last_clean":"2024-04-25T14:02:14.619161+0000","last_became_active":"2024-04-25T14:02:09.161568+0000","last_became_peered":"2024-04-25T14:02:09.161568+0000","last_unstale":"2024-04-25T14:02:14.619161+0000","last_undegraded":"2024-04-25T14:02:14.619161+0000","last_fullsized":"2024-04-25T14:02:14.619161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619443+0000","last_change":"2024-04-25T14:02:09.163237+0000","last_active":"2024-04-25T14:02:14.619443+0000","last_peered":"2024-04-25T14:02:14.619443+0000","last_clean":"2024-04-25T14:02:14.619443+0000","last_became_active":"2024-04-25T14:02:09.162870+0000","last_became_peered":"2024-04-25T14:02:09.162870+0000","last_unstale":"2024-04-25T14:02:14.619443+0000","last_undegraded":"2024-04-25T14:02:14.619443+0000","last_fullsized":"2024-04-25T14:02:14.619443+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180151+0000","last_change":"2024-04-25T14:02:09.167689+0000","last_active":"2024-04-25T14:02:14.180151+0000","last_peered":"2024-04-25T14:02:14.180151+0000","last_clean":"2024-04-25T14:02:14.180151+0000","last_became_active":"2024-04-25T14:02:09.167379+0000","last_became_peered":"2024-04-25T14:02:09.167379+0000","last_unstale":"2024-04-25T14:02:14.180151+0000","last_undegraded":"2024-04-25T14:02:14.180151+0000","last_fullsized":"2024-04-25T14:02:14.180151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621304+0000","last_change":"2024-04-25T14:02:09.155156+0000","last_active":"2024-04-25T14:02:14.621304+0000","last_peered":"2024-04-25T14:02:14.621304+0000","last_clean":"2024-04-25T14:02:14.621304+0000","last_became_active":"2024-04-25T14:02:09.154497+0000","last_became_peered":"2024-04-25T14:02:09.154497+0000","last_unstale":"2024-04-25T14:02:14.621304+0000","last_undegraded":"2024-04-25T14:02:14.621304+0000","last_fullsized":"2024-04-25T14:02:14.621304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:37:32.405111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620178+0000","last_change":"2024-04-25T14:02:09.161627+0000","last_active":"2024-04-25T14:02:14.620178+0000","last_peered":"2024-04-25T14:02:14.620178+0000","last_clean":"2024-04-25T14:02:14.620178+0000","last_became_active":"2024-04-25T14:02:09.161321+0000","last_became_peered":"2024-04-25T14:02:09.161321+0000","last_unstale":"2024-04-25T14:02:14.620178+0000","last_undegraded":"2024-04-25T14:02:14.620178+0000","last_fullsized":"2024-04-25T14:02:14.620178+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181400+0000","last_change":"2024-04-25T14:02:09.177833+0000","last_active":"2024-04-25T14:02:14.181400+0000","last_peered":"2024-04-25T14:02:14.181400+0000","last_clean":"2024-04-25T14:02:14.181400+0000","last_became_active":"2024-04-25T14:02:09.177530+0000","last_became_peered":"2024-04-25T14:02:09.177530+0000","last_unstale":"2024-04-25T14:02:14.181400+0000","last_undegraded":"2024-04-25T14:02:14.181400+0000","last_fullsized":"2024-04-25T14:02:14.181400+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.621026+0000","last_change":"2024-04-25T14:02:09.160341+0000","last_active":"2024-04-25T14:02:14.621026+0000","last_peered":"2024-04-25T14:02:14.621026+0000","last_clean":"2024-04-25T14:02:14.621026+0000","last_became_active":"2024-04-25T14:02:09.160028+0000","last_became_peered":"2024-04-25T14:02:09.160028+0000","last_unstale":"2024-04-25T14:02:14.621026+0000","last_undegraded":"2024-04-25T14:02:14.621026+0000","last_fullsized":"2024-04-25T14:02:14.621026+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619081+0000","last_change":"2024-04-25T14:02:09.163936+0000","last_active":"2024-04-25T14:02:14.619081+0000","last_peered":"2024-04-25T14:02:14.619081+0000","last_clean":"2024-04-25T14:02:14.619081+0000","last_became_active":"2024-04-25T14:02:09.163303+0000","last_became_peered":"2024-04-25T14:02:09.163303+0000","last_unstale":"2024-04-25T14:02:14.619081+0000","last_undegraded":"2024-04-25T14:02:14.619081+0000","last_fullsized":"2024-04-25T14:02:14.619081+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:47.973343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619413+0000","last_change":"2024-04-25T14:02:09.154839+0000","last_active":"2024-04-25T14:02:14.619413+0000","last_peered":"2024-04-25T14:02:14.619413+0000","last_clean":"2024-04-25T14:02:14.619413+0000","last_became_active":"2024-04-25T14:02:09.154443+0000","last_became_peered":"2024-04-25T14:02:09.154443+0000","last_unstale":"2024-04-25T14:02:14.619413+0000","last_undegraded":"2024-04-25T14:02:14.619413+0000","last_fullsized":"2024-04-25T14:02:14.619413+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:31:44.570135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619286+0000","last_change":"2024-04-25T14:02:09.171782+0000","last_active":"2024-04-25T14:02:14.619286+0000","last_peered":"2024-04-25T14:02:14.619286+0000","last_clean":"2024-04-25T14:02:14.619286+0000","last_became_active":"2024-04-25T14:02:09.171158+0000","last_became_peered":"2024-04-25T14:02:09.171158+0000","last_unstale":"2024-04-25T14:02:14.619286+0000","last_undegraded":"2024-04-25T14:02:14.619286+0000","last_fullsized":"2024-04-25T14:02:14.619286+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179956+0000","last_change":"2024-04-25T14:02:09.177793+0000","last_active":"2024-04-25T14:02:14.179956+0000","last_peered":"2024-04-25T14:02:14.179956+0000","last_clean":"2024-04-25T14:02:14.179956+0000","last_became_active":"2024-04-25T14:02:09.177475+0000","last_became_peered":"2024-04-25T14:02:09.177475+0000","last_unstale":"2024-04-25T14:02:14.179956+0000","last_undegraded":"2024-04-25T14:02:14.179956+0000","last_fullsized":"2024-04-25T14:02:14.179956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181451+0000","last_change":"2024-04-25T14:02:09.166782+0000","last_active":"2024-04-25T14:02:14.181451+0000","last_peered":"2024-04-25T14:02:14.181451+0000","last_clean":"2024-04-25T14:02:14.181451+0000","last_became_active":"2024-04-25T14:02:09.166277+0000","last_became_peered":"2024-04-25T14:02:09.166277+0000","last_unstale":"2024-04-25T14:02:14.181451+0000","last_undegraded":"2024-04-25T14:02:14.181451+0000","last_fullsized":"2024-04-25T14:02:14.181451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181472+0000","last_change":"2024-04-25T14:02:09.167448+0000","last_active":"2024-04-25T14:02:14.181472+0000","last_peered":"2024-04-25T14:02:14.181472+0000","last_clean":"2024-04-25T14:02:14.181472+0000","last_became_active":"2024-04-25T14:02:09.167136+0000","last_became_peered":"2024-04-25T14:02:09.167136+0000","last_unstale":"2024-04-25T14:02:14.181472+0000","last_undegraded":"2024-04-25T14:02:14.181472+0000","last_fullsized":"2024-04-25T14:02:14.181472+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:17:14.168620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620713+0000","last_change":"2024-04-25T14:02:09.167572+0000","last_active":"2024-04-25T14:02:14.620713+0000","last_peered":"2024-04-25T14:02:14.620713+0000","last_clean":"2024-04-25T14:02:14.620713+0000","last_became_active":"2024-04-25T14:02:09.167227+0000","last_became_peered":"2024-04-25T14:02:09.167227+0000","last_unstale":"2024-04-25T14:02:14.620713+0000","last_undegraded":"2024-04-25T14:02:14.620713+0000","last_fullsized":"2024-04-25T14:02:14.620713+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:17:38.307579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620673+0000","last_change":"2024-04-25T14:02:09.174321+0000","last_active":"2024-04-25T14:02:14.620673+0000","last_peered":"2024-04-25T14:02:14.620673+0000","last_clean":"2024-04-25T14:02:14.620673+0000","last_became_active":"2024-04-25T14:02:09.174015+0000","last_became_peered":"2024-04-25T14:02:09.174015+0000","last_unstale":"2024-04-25T14:02:14.620673+0000","last_undegraded":"2024-04-25T14:02:14.620673+0000","last_fullsized":"2024-04-25T14:02:14.620673+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.618957+0000","last_change":"2024-04-25T14:02:09.169554+0000","last_active":"2024-04-25T14:02:14.618957+0000","last_peered":"2024-04-25T14:02:14.618957+0000","last_clean":"2024-04-25T14:02:14.618957+0000","last_became_active":"2024-04-25T14:02:09.168555+0000","last_became_peered":"2024-04-25T14:02:09.168555+0000","last_unstale":"2024-04-25T14:02:14.618957+0000","last_undegraded":"2024-04-25T14:02:14.618957+0000","last_fullsized":"2024-04-25T14:02:14.618957+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619417+0000","last_change":"2024-04-25T14:02:09.160170+0000","last_active":"2024-04-25T14:02:14.619417+0000","last_peered":"2024-04-25T14:02:14.619417+0000","last_clean":"2024-04-25T14:02:14.619417+0000","last_became_active":"2024-04-25T14:02:09.159871+0000","last_became_peered":"2024-04-25T14:02:09.159871+0000","last_unstale":"2024-04-25T14:02:14.619417+0000","last_undegraded":"2024-04-25T14:02:14.619417+0000","last_fullsized":"2024-04-25T14:02:14.619417+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.179980+0000","last_change":"2024-04-25T14:02:09.159887+0000","last_active":"2024-04-25T14:02:14.179980+0000","last_peered":"2024-04-25T14:02:14.179980+0000","last_clean":"2024-04-25T14:02:14.179980+0000","last_became_active":"2024-04-25T14:02:09.159431+0000","last_became_peered":"2024-04-25T14:02:09.159431+0000","last_unstale":"2024-04-25T14:02:14.179980+0000","last_undegraded":"2024-04-25T14:02:14.179980+0000","last_fullsized":"2024-04-25T14:02:14.179980+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620557+0000","last_change":"2024-04-25T14:02:09.167290+0000","last_active":"2024-04-25T14:02:14.620557+0000","last_peered":"2024-04-25T14:02:14.620557+0000","last_clean":"2024-04-25T14:02:14.620557+0000","last_became_active":"2024-04-25T14:02:09.166974+0000","last_became_peered":"2024-04-25T14:02:09.166974+0000","last_unstale":"2024-04-25T14:02:14.620557+0000","last_undegraded":"2024-04-25T14:02:14.620557+0000","last_fullsized":"2024-04-25T14:02:14.620557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.620085+0000","last_change":"2024-04-25T14:02:09.169185+0000","last_active":"2024-04-25T14:02:14.620085+0000","last_peered":"2024-04-25T14:02:14.620085+0000","last_clean":"2024-04-25T14:02:14.620085+0000","last_became_active":"2024-04-25T14:02:09.168862+0000","last_became_peered":"2024-04-25T14:02:09.168862+0000","last_unstale":"2024-04-25T14:02:14.620085+0000","last_undegraded":"2024-04-25T14:02:14.620085+0000","last_fullsized":"2024-04-25T14:02:14.620085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:11:39.506068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180614+0000","last_change":"2024-04-25T14:02:09.174006+0000","last_active":"2024-04-25T14:02:14.180614+0000","last_peered":"2024-04-25T14:02:14.180614+0000","last_clean":"2024-04-25T14:02:14.180614+0000","last_became_active":"2024-04-25T14:02:09.173662+0000","last_became_peered":"2024-04-25T14:02:09.173662+0000","last_unstale":"2024-04-25T14:02:14.180614+0000","last_undegraded":"2024-04-25T14:02:14.180614+0000","last_fullsized":"2024-04-25T14:02:14.180614+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:28:29.479208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.181263+0000","last_change":"2024-04-25T14:02:09.174331+0000","last_active":"2024-04-25T14:02:14.181263+0000","last_peered":"2024-04-25T14:02:14.181263+0000","last_clean":"2024-04-25T14:02:14.181263+0000","last_became_active":"2024-04-25T14:02:09.173998+0000","last_became_peered":"2024-04-25T14:02:09.173998+0000","last_unstale":"2024-04-25T14:02:14.181263+0000","last_undegraded":"2024-04-25T14:02:14.181263+0000","last_fullsized":"2024-04-25T14:02:14.181263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:54.281702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182263+0000","last_change":"2024-04-25T14:02:09.166767+0000","last_active":"2024-04-25T14:02:14.182263+0000","last_peered":"2024-04-25T14:02:14.182263+0000","last_clean":"2024-04-25T14:02:14.182263+0000","last_became_active":"2024-04-25T14:02:09.165131+0000","last_became_peered":"2024-04-25T14:02:09.165131+0000","last_unstale":"2024-04-25T14:02:14.182263+0000","last_undegraded":"2024-04-25T14:02:14.182263+0000","last_fullsized":"2024-04-25T14:02:14.182263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:51:07.386855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619681+0000","last_change":"2024-04-25T14:02:09.164960+0000","last_active":"2024-04-25T14:02:14.619681+0000","last_peered":"2024-04-25T14:02:14.619681+0000","last_clean":"2024-04-25T14:02:14.619681+0000","last_became_active":"2024-04-25T14:02:09.164417+0000","last_became_peered":"2024-04-25T14:02:09.164417+0000","last_unstale":"2024-04-25T14:02:14.619681+0000","last_undegraded":"2024-04-25T14:02:14.619681+0000","last_fullsized":"2024-04-25T14:02:14.619681+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:13:49.624991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180423+0000","last_change":"2024-04-25T14:02:09.161177+0000","last_active":"2024-04-25T14:02:14.180423+0000","last_peered":"2024-04-25T14:02:14.180423+0000","last_clean":"2024-04-25T14:02:14.180423+0000","last_became_active":"2024-04-25T14:02:09.160404+0000","last_became_peered":"2024-04-25T14:02:09.160404+0000","last_unstale":"2024-04-25T14:02:14.180423+0000","last_undegraded":"2024-04-25T14:02:14.180423+0000","last_fullsized":"2024-04-25T14:02:14.180423+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:16.752542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619241+0000","last_change":"2024-04-25T14:02:09.170498+0000","last_active":"2024-04-25T14:02:14.619241+0000","last_peered":"2024-04-25T14:02:14.619241+0000","last_clean":"2024-04-25T14:02:14.619241+0000","last_became_active":"2024-04-25T14:02:09.167994+0000","last_became_peered":"2024-04-25T14:02:09.167994+0000","last_unstale":"2024-04-25T14:02:14.619241+0000","last_undegraded":"2024-04-25T14:02:14.619241+0000","last_fullsized":"2024-04-25T14:02:14.619241+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:18:39.723585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180565+0000","last_change":"2024-04-25T14:02:09.155470+0000","last_active":"2024-04-25T14:02:14.180565+0000","last_peered":"2024-04-25T14:02:14.180565+0000","last_clean":"2024-04-25T14:02:14.180565+0000","last_became_active":"2024-04-25T14:02:09.155123+0000","last_became_peered":"2024-04-25T14:02:09.155123+0000","last_unstale":"2024-04-25T14:02:14.180565+0000","last_undegraded":"2024-04-25T14:02:14.180565+0000","last_fullsized":"2024-04-25T14:02:14.180565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:55:48.221643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619577+0000","last_change":"2024-04-25T14:02:09.163082+0000","last_active":"2024-04-25T14:02:14.619577+0000","last_peered":"2024-04-25T14:02:14.619577+0000","last_clean":"2024-04-25T14:02:14.619577+0000","last_became_active":"2024-04-25T14:02:09.162148+0000","last_became_peered":"2024-04-25T14:02:09.162148+0000","last_unstale":"2024-04-25T14:02:14.619577+0000","last_undegraded":"2024-04-25T14:02:14.619577+0000","last_fullsized":"2024-04-25T14:02:14.619577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:01:21.732323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.180249+0000","last_change":"2024-04-25T14:02:09.167223+0000","last_active":"2024-04-25T14:02:14.180249+0000","last_peered":"2024-04-25T14:02:14.180249+0000","last_clean":"2024-04-25T14:02:14.180249+0000","last_became_active":"2024-04-25T14:02:09.166924+0000","last_became_peered":"2024-04-25T14:02:09.166924+0000","last_unstale":"2024-04-25T14:02:14.180249+0000","last_undegraded":"2024-04-25T14:02:14.180249+0000","last_fullsized":"2024-04-25T14:02:14.180249+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.619169+0000","last_change":"2024-04-25T14:02:09.159838+0000","last_active":"2024-04-25T14:02:14.619169+0000","last_peered":"2024-04-25T14:02:14.619169+0000","last_clean":"2024-04-25T14:02:14.619169+0000","last_became_active":"2024-04-25T14:02:09.159551+0000","last_became_peered":"2024-04-25T14:02:09.159551+0000","last_unstale":"2024-04-25T14:02:14.619169+0000","last_undegraded":"2024-04-25T14:02:14.619169+0000","last_fullsized":"2024-04-25T14:02:14.619169+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:29:25.767388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-04-25T14:02:14.182085+0000","last_change":"2024-04-25T14:02:09.171351+0000","last_active":"2024-04-25T14:02:14.182085+0000","last_peered":"2024-04-25T14:02:14.182085+0000","last_clean":"2024-04-25T14:02:14.182085+0000","last_became_active":"2024-04-25T14:02:09.168711+0000","last_became_peered":"2024-04-25T14:02:09.168711+0000","last_unstale":"2024-04-25T14:02:14.182085+0000","last_undegraded":"2024-04-25T14:02:14.182085+0000","last_fullsized":"2024-04-25T14:02:14.182085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:02:08.119178+0000","last_clean_scrub_stamp":"2024-04-25T14:02:08.119178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:29:35.203858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574860,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":851968,"data_stored":691206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574860,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574860,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":100382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":266240,"data_stored":104098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574859,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27276,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344564,"statfs":{"total":96636764160,"available":96608833536,"internally_reserved":0,"allocated":253952,"data_stored":100256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":691262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:02:37.185 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T14:02:37.185 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T14:02:37.185 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-25T14:02:37.185 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-25T14:02:37.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.328+0000 7f5d77642640 1 Processor -- start 2024-04-25T14:02:37.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.328+0000 7f5d77642640 1 -- start start 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d77642640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5d700b21c0 0x7f5d7006f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d77642640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d700b28b0 con 0x7f5d700d04e0 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d77642640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d700b2b90 con 0x7f5d70160db0 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d77642640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5d7006f770 con 0x7f5d700b21c0 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d74bdd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5d700d04e0 0x7f5d701606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55678/0 (socket says 172.21.15.150:55678) 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.329+0000 7f5d74bdd640 1 -- 172.21.15.150:0/1995997037 learned_addr learned my addr 172.21.15.150:0/1995997037 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:37.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d753de640 1 --2- 172.21.15.150:0/1995997037 >> v2:172.21.15.187:3300/0 conn(0x7f5d700b21c0 0x7f5d7006f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3583190662 0 0) 0x7f5d700b28b0 con 0x7f5d700d04e0 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d5c003520 con 0x7f5d700d04e0 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3933129419 0 0) 0x7f5d700b2b90 con 0x7f5d70160db0 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d700b28b0 con 0x7f5d70160db0 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d753de640 1 -- 172.21.15.150:0/1995997037 >> v1:172.21.15.187:6789/0 conn(0x7f5d70160db0 legacy=0x7f5d700b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d753de640 1 -- 172.21.15.150:0/1995997037 >> v1:172.21.15.150:6789/0 conn(0x7f5d700d04e0 legacy=0x7f5d701606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.330+0000 7f5d753de640 1 -- 172.21.15.150:0/1995997037 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d700b2e70 con 0x7f5d700b21c0 2024-04-25T14:02:37.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.331+0000 7f5d753de640 1 --2- 172.21.15.150:0/1995997037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700b21c0 0x7f5d7006f230 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009790 tx=0x7f5d6c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37cb7c904190afcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.331+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c037070 con 0x7f5d700b21c0 2024-04-25T14:02:37.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.331+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c032440 con 0x7f5d700b21c0 2024-04-25T14:02:37.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.331+0000 7f5d67fff640 1 -- 172.21.15.150:0/1995997037 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c032810 con 0x7f5d700b21c0 2024-04-25T14:02:37.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 -- 172.21.15.150:0/1995997037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700b21c0 msgr2=0x7f5d7006f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 --2- 172.21.15.150:0/1995997037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700b21c0 0x7f5d7006f230 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c009790 tx=0x7f5d6c02a180 comp rx=0 tx=0).stop 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 -- 172.21.15.150:0/1995997037 shutdown_connections 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 --2- 172.21.15.150:0/1995997037 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700b21c0 0x7f5d7006f230 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 -- 172.21.15.150:0/1995997037 >> 172.21.15.150:0/1995997037 conn(0x7f5d7009b4a0 msgr2=0x7f5d7009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 -- 172.21.15.150:0/1995997037 shutdown_connections 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.332+0000 7f5d77642640 1 -- 172.21.15.150:0/1995997037 wait complete. 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.333+0000 7f5d77642640 1 Processor -- start 2024-04-25T14:02:37.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.333+0000 7f5d77642640 1 -- start start 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700d04e0 0x7f5d70153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d70160db0 0x7f5d700b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5d700d08f0 con 0x7f5d700b21c0 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d700b25d0 con 0x7f5d700d04e0 2024-04-25T14:02:37.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d77642640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d700d3030 con 0x7f5d70160db0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:38660/0 (socket says 172.21.15.150:38660) 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 -- 172.21.15.150:0/3718380863 learned_addr learned my addr 172.21.15.150:0/3718380863 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d74bdd640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700d04e0 0x7f5d70153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d75bdf640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d70160db0 0x7f5d700b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 -- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d70160db0 msgr2=0x7f5d700b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d70160db0 0x7f5d700b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 -- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700d04e0 msgr2=0x7f5d70153b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700d04e0 0x7f5d70153b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 -- 172.21.15.150:0/3718380863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d700afd00 con 0x7f5d700b21c0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.335+0000 7f5d753de640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c02a1c0 tx=0x7f5d6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.336+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c037040 con 0x7f5d700b21c0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.336+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d6c042070 con 0x7f5d700b21c0 2024-04-25T14:02:37.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.336+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d700affe0 con 0x7f5d700b21c0 2024-04-25T14:02:37.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.336+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d6c03b950 con 0x7f5d700b21c0 2024-04-25T14:02:37.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.336+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d700b05a0 con 0x7f5d700b21c0 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.337+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d70076a40 con 0x7f5d700b21c0 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.338+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d6c03baf0 con 0x7f5d700b21c0 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.338+0000 7f5d65ffb640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 0x7f5d540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.338+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5d6c0b5ae0 con 0x7f5d700b21c0 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.338+0000 7f5d74bdd640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 0x7f5d540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.338+0000 7f5d74bdd640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 0x7f5d540769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d60000f60 tx=0x7f5d60007d70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.343+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d6c07f970 con 0x7f5d700b21c0 2024-04-25T14:02:37.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.559+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f5d7006c650 con 0x7f5d700b21c0 2024-04-25T14:02:37.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.560+0000 7f5d65ffb640 1 -- 172.21.15.150:0/3718380863 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f5d6c083c70 con 0x7f5d700b21c0 2024-04-25T14:02:37.559 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:37.559 INFO:teuthology.orchestra.run.smithi150.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T14:02:37.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 msgr2=0x7f5d540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 0x7f5d540769c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5d60000f60 tx=0x7f5d60007d70 comp rx=0 tx=0).stop 2024-04-25T14:02:37.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 msgr2=0x7f5d700b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5d6c02a1c0 tx=0x7f5d6c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 shutdown_connections 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d540744b0 0x7f5d540769c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d70160db0 0x7f5d700b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d700d04e0 0x7f5d70153b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 --2- 172.21.15.150:0/3718380863 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d700b21c0 0x7f5d700b1510 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 >> 172.21.15.150:0/3718380863 conn(0x7f5d7009b4a0 msgr2=0x7f5d7015fe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 shutdown_connections 2024-04-25T14:02:37.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.562+0000 7f5d77642640 1 -- 172.21.15.150:0/3718380863 wait complete. 2024-04-25T14:02:37.572 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-25T14:02:37.573 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:37.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.681+0000 7ffba531f640 1 Processor -- start 2024-04-25T14:02:37.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.681+0000 7ffba531f640 1 -- start start 2024-04-25T14:02:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffba531f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ffba01608e0 0x7ffba0160cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffba531f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffba0161230 0x7ffba0169cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffba531f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba006e040 con 0x7ffba00d4470 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffba531f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ffba015abb0 con 0x7ffba0161230 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffba531f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ffba006c570 con 0x7ffba01608e0 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffba0161230 0x7ffba0169cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ffba0161230 0x7ffba0169cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56312/0 (socket says 172.21.15.150:56312) 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 -- 172.21.15.150:0/2171007234 learned_addr learned my addr 172.21.15.150:0/2171007234 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/2171007234 >> v2:172.21.15.187:3300/0 conn(0x7ffba01608e0 0x7ffba0160cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 -- 172.21.15.150:0/2171007234 >> v2:172.21.15.187:3300/0 conn(0x7ffba01608e0 msgr2=0x7ffba0160cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 --2- 172.21.15.150:0/2171007234 >> v2:172.21.15.187:3300/0 conn(0x7ffba01608e0 0x7ffba0160cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 -- 172.21.15.150:0/2171007234 >> v1:172.21.15.187:6789/0 conn(0x7ffba00d4470 legacy=0x7ffba00d4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.683+0000 7ffb9f7fe640 1 -- 172.21.15.150:0/2171007234 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba00d5130 con 0x7ffba0161230 2024-04-25T14:02:37.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.684+0000 7ffb9f7fe640 1 --2- 172.21.15.150:0/2171007234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0161230 0x7ffba0169cb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffba008e760 tx=0x7ffb9802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1278a2ff924830f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.684+0000 7ffb9dffb640 1 -- 172.21.15.150:0/2171007234 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb98037070 con 0x7ffba0161230 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.684+0000 7ffb9dffb640 1 -- 172.21.15.150:0/2171007234 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb98032440 con 0x7ffba0161230 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.684+0000 7ffb9dffb640 1 -- 172.21.15.150:0/2171007234 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb98032810 con 0x7ffba0161230 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 -- 172.21.15.150:0/2171007234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0161230 msgr2=0x7ffba0169cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 --2- 172.21.15.150:0/2171007234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0161230 0x7ffba0169cb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ffba008e760 tx=0x7ffb9802a270 comp rx=0 tx=0).stop 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 -- 172.21.15.150:0/2171007234 shutdown_connections 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 --2- 172.21.15.150:0/2171007234 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba0161230 0x7ffba0169cb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 --2- 172.21.15.150:0/2171007234 >> v2:172.21.15.187:3300/0 conn(0x7ffba01608e0 0x7ffba0160cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 -- 172.21.15.150:0/2171007234 >> 172.21.15.150:0/2171007234 conn(0x7ffba00d2590 msgr2=0x7ffba00d29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 -- 172.21.15.150:0/2171007234 shutdown_connections 2024-04-25T14:02:37.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.685+0000 7ffba531f640 1 -- 172.21.15.150:0/2171007234 wait complete. 2024-04-25T14:02:37.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.687+0000 7ffba531f640 1 Processor -- start 2024-04-25T14:02:37.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.687+0000 7ffba531f640 1 -- start start 2024-04-25T14:02:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffba01608e0 0x7ffba01eb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffba0161230 0x7ffba01f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffba006cb10 con 0x7ffba00d4470 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffba006c830 con 0x7ffba01608e0 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffba531f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffba0161850 con 0x7ffba0161230 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffb9effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffb9effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56314/0 (socket says 172.21.15.150:56314) 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffb9effd640 1 -- 172.21.15.150:0/526306669 learned_addr learned my addr 172.21.15.150:0/526306669 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffba01608e0 0x7ffba01eb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.688+0000 7ffb9f7fe640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffba0161230 0x7ffba01f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 -- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffba0161230 msgr2=0x7ffba01f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffba0161230 0x7ffba01f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 -- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffba01608e0 msgr2=0x7ffba01eb150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffba01608e0 0x7ffba01eb150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 -- 172.21.15.150:0/526306669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba01e9b60 con 0x7ffba00d4470 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb9effd640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c6f0 tx=0x7ffb9000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb90003990 con 0x7ffba00d4470 2024-04-25T14:02:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb90003b50 con 0x7ffba00d4470 2024-04-25T14:02:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba01e9e40 con 0x7ffba00d4470 2024-04-25T14:02:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000a850 con 0x7ffba00d4470 2024-04-25T14:02:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.689+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba01ea400 con 0x7ffba00d4470 2024-04-25T14:02:37.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.690+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba0076a40 con 0x7ffba00d4470 2024-04-25T14:02:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.691+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffb9000a9f0 con 0x7ffba00d4470 2024-04-25T14:02:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.691+0000 7ffb877fe640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 0x7ffb8c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.691+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 0x7ffb8c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.691+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7ffb90095360 con 0x7ffba00d4470 2024-04-25T14:02:37.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.692+0000 7ffb9e7fc640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 0x7ffb8c0769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffba0158d10 tx=0x7ffb94009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:37.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.696+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffb9005f0e0 con 0x7ffba00d4470 2024-04-25T14:02:37.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.870+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffba00d58d0 con 0x7ffba00d4470 2024-04-25T14:02:37.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.872+0000 7ffb877fe640 1 -- 172.21.15.150:0/526306669 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7ffb900633e0 con 0x7ffba00d4470 2024-04-25T14:02:37.871 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:37.871 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:37.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 msgr2=0x7ffb8c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 0x7ffb8c0769c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffba0158d10 tx=0x7ffb94009190 comp rx=0 tx=0).stop 2024-04-25T14:02:37.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 msgr2=0x7ffba01e9310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c6f0 tx=0x7ffb9000cbb0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 shutdown_connections 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffb8c0744b0 0x7ffb8c0769c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffba0161230 0x7ffba01f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffba01608e0 0x7ffba01eb150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 --2- 172.21.15.150:0/526306669 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffba00d4470 0x7ffba01e9310 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 >> 172.21.15.150:0/526306669 conn(0x7ffba00d2590 msgr2=0x7ffba00bb030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 shutdown_connections 2024-04-25T14:02:37.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.874+0000 7ffba531f640 1 -- 172.21.15.150:0/526306669 wait complete. 2024-04-25T14:02:37.885 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:37.886 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:37.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.996+0000 7f147fad1640 1 Processor -- start 2024-04-25T14:02:37.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.996+0000 7f147fad1640 1 -- start start 2024-04-25T14:02:37.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147fad1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14780b5ad0 0x7f147806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147fad1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:37.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147fad1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f147806ff50 con 0x7f1478160460 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147fad1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f147806f3d0 con 0x7f14780b5ad0 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147fad1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f147806f6b0 con 0x7f1478163d80 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d06c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14780b5ad0 0x7f147806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d86d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f1478160460 0x7f14780b53c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41198/0 (socket says 172.21.15.150:41198) 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d86d640 1 -- 172.21.15.150:0/3619728370 learned_addr learned my addr 172.21.15.150:0/3619728370 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147e06e640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d06c640 1 -- 172.21.15.150:0/3619728370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 msgr2=0x7f14780d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d06c640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d06c640 1 -- 172.21.15.150:0/3619728370 >> v1:172.21.15.150:6789/0 conn(0x7f1478160460 legacy=0x7f14780b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147d06c640 1 -- 172.21.15.150:0/3619728370 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147806f990 con 0x7f14780b5ad0 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147c86b640 1 -- 172.21.15.150:0/3619728370 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186361801 0 0) 0x7f147806ff50 con 0x7f1478160460 2024-04-25T14:02:37.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.998+0000 7f147e06e640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.999+0000 7f147d06c640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14780b5ad0 0x7f147806ed40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f14680097a0 tx=0x7f146802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=366d658a3695f4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.999+0000 7f147c86b640 1 -- 172.21.15.150:0/3619728370 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468037070 con 0x7f14780b5ad0 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.999+0000 7f147c86b640 1 -- 172.21.15.150:0/3619728370 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f146802ae50 con 0x7f14780b5ad0 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:37.999+0000 7f147c86b640 1 -- 172.21.15.150:0/3619728370 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1468032660 con 0x7f14780b5ad0 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 -- 172.21.15.150:0/3619728370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14780b5ad0 msgr2=0x7f147806ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14780b5ad0 0x7f147806ed40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f14680097a0 tx=0x7f146802a100 comp rx=0 tx=0).stop 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 -- 172.21.15.150:0/3619728370 shutdown_connections 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 --2- 172.21.15.150:0/3619728370 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14780b5ad0 0x7f147806ed40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 -- 172.21.15.150:0/3619728370 >> 172.21.15.150:0/3619728370 conn(0x7f14780d03c0 msgr2=0x7f14780d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 -- 172.21.15.150:0/3619728370 shutdown_connections 2024-04-25T14:02:38.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.000+0000 7f147fad1640 1 -- 172.21.15.150:0/3619728370 wait complete. 2024-04-25T14:02:38.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.002+0000 7f147fad1640 1 Processor -- start 2024-04-25T14:02:38.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.002+0000 7f147fad1640 1 -- start start 2024-04-25T14:02:38.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b82a0 0x7f147815aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14780ce5c0 con 0x7f147815b430 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14780b3030 con 0x7f14780b82a0 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147fad1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14780d1da0 con 0x7f1478163d80 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56330/0 (socket says 172.21.15.150:56330) 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 -- 172.21.15.150:0/2159373147 learned_addr learned my addr 172.21.15.150:0/2159373147 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147d06c640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b82a0 0x7f147815aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147d86d640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 -- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 msgr2=0x7f14780b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 -- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b82a0 msgr2=0x7f147815aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b82a0 0x7f147815aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.003+0000 7f147e06e640 1 -- 172.21.15.150:0/2159373147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147815b840 con 0x7f147815b430 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f147e06e640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f147000c7c0 tx=0x7f147000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470015020 con 0x7f147815b430 2024-04-25T14:02:38.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1470004230 con 0x7f147815b430 2024-04-25T14:02:38.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1470007620 con 0x7f147815b430 2024-04-25T14:02:38.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1478160f10 con 0x7f147815b430 2024-04-25T14:02:38.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.004+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14781614d0 con 0x7f147815b430 2024-04-25T14:02:38.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.005+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1478076a40 con 0x7f147815b430 2024-04-25T14:02:38.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.005+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1470018020 con 0x7f147815b430 2024-04-25T14:02:38.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.006+0000 7f146e7fc640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 0x7f145c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.006+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1470095450 con 0x7f147815b430 2024-04-25T14:02:38.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.006+0000 7f147d86d640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 0x7f145c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.006+0000 7f147d86d640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 0x7f145c0769c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14740012e0 tx=0x7f1474001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.010+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f147001f020 con 0x7f147815b430 2024-04-25T14:02:38.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.184+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14780b8740 con 0x7f147815b430 2024-04-25T14:02:38.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.185+0000 7f146e7fc640 1 -- 172.21.15.150:0/2159373147 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f147815b840 con 0x7f147815b430 2024-04-25T14:02:38.184 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.186+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 msgr2=0x7f145c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 0x7f145c0769c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f14740012e0 tx=0x7f1474001630 comp rx=0 tx=0).stop 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 msgr2=0x7f14780b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f147000c7c0 tx=0x7f147000cc80 comp rx=0 tx=0).stop 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 shutdown_connections 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f145c0744b0 0x7f145c0769c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f147815b430 0x7f14780b6560 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14780b82a0 0x7f147815aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 --2- 172.21.15.150:0/2159373147 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1478163d80 0x7f14780b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 >> 172.21.15.150:0/2159373147 conn(0x7f14780d03c0 msgr2=0x7f147815d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 shutdown_connections 2024-04-25T14:02:38.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.187+0000 7f147fad1640 1 -- 172.21.15.150:0/2159373147 wait complete. 2024-04-25T14:02:38.197 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:38.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:02:38.266 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:02:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.312+0000 7f92c01f3640 1 Processor -- start 2024-04-25T14:02:38.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.312+0000 7f92c01f3640 1 -- start start 2024-04-25T14:02:38.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.313+0000 7f92c01f3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 0x7f92b8158700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.313+0000 7f92c01f3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 0x7f92b8167cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.313+0000 7f92c01f3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b815eb60 con 0x7f92b809b4a0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.313+0000 7f92c01f3640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f92b806e7a0 con 0x7f92b815e6a0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.313+0000 7f92c01f3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92b815ee40 con 0x7f92b80b42a0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bd78e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f92b809b4a0 0x7f92b80b3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41208/0 (socket says 172.21.15.150:41208) 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bd78e640 1 -- 172.21.15.150:0/3881373229 learned_addr learned my addr 172.21.15.150:0/3881373229 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bdf8f640 1 --2- 172.21.15.150:0/3881373229 >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 0x7f92b8158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92be790640 1 --2- 172.21.15.150:0/3881373229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 0x7f92b8167cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454230697 0 0) 0x7f92b815eb60 con 0x7f92b809b4a0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a4003520 con 0x7f92b809b4a0 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92be790640 1 -- 172.21.15.150:0/3881373229 >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 msgr2=0x7f92b8158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92be790640 1 --2- 172.21.15.150:0/3881373229 >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 0x7f92b8158700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92be790640 1 -- 172.21.15.150:0/3881373229 >> v1:172.21.15.150:6789/0 conn(0x7f92b809b4a0 legacy=0x7f92b80b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4028785489 0 0) 0x7f92a4003520 con 0x7f92b809b4a0 2024-04-25T14:02:38.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92be790640 1 -- 172.21.15.150:0/3881373229 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b815f120 con 0x7f92b80b42a0 2024-04-25T14:02:38.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.314+0000 7f92bdf8f640 1 --2- 172.21.15.150:0/3881373229 >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 0x7f92b8158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.315+0000 7f92be790640 1 --2- 172.21.15.150:0/3881373229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 0x7f92b8167cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009d90 tx=0x7f92b002a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad0229989debce3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.315+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b4001670 con 0x7f92b80b42a0 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.315+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b0007540 con 0x7f92b80b42a0 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.315+0000 7f92bcf8d640 1 -- 172.21.15.150:0/3881373229 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b00078f0 con 0x7f92b80b42a0 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 -- 172.21.15.150:0/3881373229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 msgr2=0x7f92b8167cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 --2- 172.21.15.150:0/3881373229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 0x7f92b8167cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f92b0009d90 tx=0x7f92b002a560 comp rx=0 tx=0).stop 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 -- 172.21.15.150:0/3881373229 shutdown_connections 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 --2- 172.21.15.150:0/3881373229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b80b42a0 0x7f92b8167cc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 --2- 172.21.15.150:0/3881373229 >> v2:172.21.15.187:3300/0 conn(0x7f92b815e6a0 0x7f92b8158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 -- 172.21.15.150:0/3881373229 >> 172.21.15.150:0/3881373229 conn(0x7f92b8073d50 msgr2=0x7f92b80b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 -- 172.21.15.150:0/3881373229 shutdown_connections 2024-04-25T14:02:38.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.316+0000 7f92c01f3640 1 -- 172.21.15.150:0/3881373229 wait complete. 2024-04-25T14:02:38.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.318+0000 7f92c01f3640 1 Processor -- start 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.318+0000 7f92c01f3640 1 -- start start 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b8151430 0x7f92b814e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92b809cd20 0x7f92b809d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f92b809ba30 con 0x7f92b809b4a0 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f92b809b8b0 con 0x7f92b809cd20 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92c01f3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92b8160c30 con 0x7f92b8151430 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56342/0 (socket says 172.21.15.150:56342) 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 -- 172.21.15.150:0/2079804060 learned_addr learned my addr 172.21.15.150:0/2079804060 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92be790640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92b809cd20 0x7f92b809d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bd78e640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b8151430 0x7f92b814e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 -- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b8151430 msgr2=0x7f92b814e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b8151430 0x7f92b814e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 -- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92b809cd20 msgr2=0x7f92b809d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92b809cd20 0x7f92b809d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.319+0000 7f92bdf8f640 1 -- 172.21.15.150:0/2079804060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b8151840 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92bdf8f640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e940 tx=0x7f92a800ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8017070 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92a800ca00 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b81537b0 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a800cdd0 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.320+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b809c240 con 0x7f92b809b4a0 2024-04-25T14:02:38.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.321+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b8076a40 con 0x7f92b809b4a0 2024-04-25T14:02:38.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.321+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f92a801a070 con 0x7f92b809b4a0 2024-04-25T14:02:38.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.322+0000 7f92ae7fc640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 0x7f929c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.322+0000 7f92bd78e640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 0x7f929c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.322+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f92a80953b0 con 0x7f92b809b4a0 2024-04-25T14:02:38.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.322+0000 7f92bd78e640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 0x7f929c0769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f92b81f7d40 tx=0x7f92b4001d80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.327+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92a805f240 con 0x7f92b809b4a0 2024-04-25T14:02:38.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.501+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f92b80b14c0 con 0x7f92b809b4a0 2024-04-25T14:02:38.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.501+0000 7f92ae7fc640 1 -- 172.21.15.150:0/2079804060 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7f92a8063540 con 0x7f92b809b4a0 2024-04-25T14:02:38.500 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 msgr2=0x7f929c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 0x7f929c0769c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f92b81f7d40 tx=0x7f92b4001d80 comp rx=0 tx=0).stop 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 msgr2=0x7f92b80d2a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e940 tx=0x7f92a800ee00 comp rx=0 tx=0).stop 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 shutdown_connections 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f929c0744b0 0x7f929c0769c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92b809cd20 0x7f92b809d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92b8151430 0x7f92b814e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 --2- 172.21.15.150:0/2079804060 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92b809b4a0 0x7f92b80d2a50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 >> 172.21.15.150:0/2079804060 conn(0x7f92b8073d50 msgr2=0x7f92b80b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 shutdown_connections 2024-04-25T14:02:38.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.503+0000 7f92c01f3640 1 -- 172.21.15.150:0/2079804060 wait complete. 2024-04-25T14:02:38.514 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:38.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.668+0000 7f47ae8d9640 1 Processor -- start 2024-04-25T14:02:38.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.668+0000 7f47ae8d9640 1 -- start start 2024-04-25T14:02:38.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ae8d9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a80735b0 0x7f47a8160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ae8d9640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f47a80b6800 0x7f47a80b8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ae8d9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47a8164250 con 0x7f47a80d3280 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ae8d9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f47a81644f0 con 0x7f47a80735b0 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ae8d9640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f47a81647d0 con 0x7f47a80b6800 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a80735b0 0x7f47a8160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47a7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f47a80d3280 0x7f47a8072ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41218/0 (socket says 172.21.15.150:41218) 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47a77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f47a80735b0 0x7f47a8160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56346/0 (socket says 172.21.15.150:56346) 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47a7fff640 1 -- 172.21.15.150:0/163545126 learned_addr learned my addr 172.21.15.150:0/163545126 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.669+0000 7f47ace76640 1 --2- 172.21.15.150:0/163545126 >> v2:172.21.15.187:3300/0 conn(0x7f47a80b6800 0x7f47a80b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a77fe640 1 -- 172.21.15.150:0/163545126 >> v2:172.21.15.187:3300/0 conn(0x7f47a80b6800 msgr2=0x7f47a80b8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a77fe640 1 --2- 172.21.15.150:0/163545126 >> v2:172.21.15.187:3300/0 conn(0x7f47a80b6800 0x7f47a80b8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a77fe640 1 -- 172.21.15.150:0/163545126 >> v1:172.21.15.150:6789/0 conn(0x7f47a80d3280 legacy=0x7f47a8072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a77fe640 1 -- 172.21.15.150:0/163545126 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a8164ab0 con 0x7f47a80735b0 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a6ffd640 1 -- 172.21.15.150:0/163545126 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3757872253 0 0) 0x7f47a8164250 con 0x7f47a80d3280 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.670+0000 7f47a77fe640 1 --2- 172.21.15.150:0/163545126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80735b0 0x7f47a8160360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47a80d3960 tx=0x7f479c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424f6055493ea22f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47a6ffd640 1 -- 172.21.15.150:0/163545126 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c037070 con 0x7f47a80735b0 2024-04-25T14:02:38.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47a6ffd640 1 -- 172.21.15.150:0/163545126 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f479c032440 con 0x7f47a80735b0 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47a6ffd640 1 -- 172.21.15.150:0/163545126 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c032810 con 0x7f47a80735b0 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 -- 172.21.15.150:0/163545126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80735b0 msgr2=0x7f47a8160360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/163545126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80735b0 0x7f47a8160360 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f47a80d3960 tx=0x7f479c02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 -- 172.21.15.150:0/163545126 shutdown_connections 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/163545126 >> v2:172.21.15.187:3300/0 conn(0x7f47a80b6800 0x7f47a80b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/163545126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80735b0 0x7f47a8160360 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 -- 172.21.15.150:0/163545126 >> 172.21.15.150:0/163545126 conn(0x7f47a80d39b0 msgr2=0x7f47a80d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 -- 172.21.15.150:0/163545126 shutdown_connections 2024-04-25T14:02:38.675 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.671+0000 7f47ae8d9640 1 -- 172.21.15.150:0/163545126 wait complete. 2024-04-25T14:02:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.673+0000 7f47ae8d9640 1 Processor -- start 2024-04-25T14:02:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.673+0000 7f47ae8d9640 1 -- start start 2024-04-25T14:02:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.674+0000 7f47ae8d9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47a80b6800 0x7f47a81623b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 0x7f47a80ce180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f47a80ce000 con 0x7f47a80700e0 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47a8163f40 con 0x7f47a80d3280 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47a80b89a0 con 0x7f47a80b6800 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ace76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ace76640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56348/0 (socket says 172.21.15.150:56348) 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ace76640 1 -- 172.21.15.150:0/2391108499 learned_addr learned my addr 172.21.15.150:0/2391108499 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 0x7f47a80ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a7fff640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47a80b6800 0x7f47a81623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 -- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47a80b6800 msgr2=0x7f47a81623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47a80b6800 0x7f47a81623b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 -- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 msgr2=0x7f47a80ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 -- 172.21.15.150:0/2391108499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47a8162be0 con 0x7f47a80d3280 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ace76640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a77fe640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 0x7f47a80ce180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f479c031000 tx=0x7f479c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c041020 con 0x7f47a80d3280 2024-04-25T14:02:38.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f479c030720 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47a8162ec0 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.675+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f479c037040 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.676+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47a80b3490 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.676+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47a8076a40 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.677+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f479c044020 con 0x7f47a80d3280 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.677+0000 7f47a4ff9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 0x7f47940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.678+0000 7f47a7fff640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 0x7f47940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.678+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f479c0b5950 con 0x7f47a80d3280 2024-04-25T14:02:38.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.678+0000 7f47a7fff640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 0x7f47940769f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4798007fd0 tx=0x7f4798007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.682+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f479c04b030 con 0x7f47a80d3280 2024-04-25T14:02:38.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.856+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f47a8162970 con 0x7f47a80d3280 2024-04-25T14:02:38.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.857+0000 7f47a4ff9640 1 -- 172.21.15.150:0/2391108499 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f479c07f740 con 0x7f47a80d3280 2024-04-25T14:02:38.857 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 msgr2=0x7f47940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 0x7f47940769f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4798007fd0 tx=0x7f4798007de0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 msgr2=0x7f47a80ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 0x7f47a80ce180 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f479c031000 tx=0x7f479c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 shutdown_connections 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47940744e0 0x7f47940769f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47a80700e0 0x7f47a80ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47a80d3280 0x7f47a80ce180 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 --2- 172.21.15.150:0/2391108499 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47a80b6800 0x7f47a81623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 >> 172.21.15.150:0/2391108499 conn(0x7f47a80d39b0 msgr2=0x7f47a815c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 shutdown_connections 2024-04-25T14:02:38.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.859+0000 7f47ae8d9640 1 -- 172.21.15.150:0/2391108499 wait complete. 2024-04-25T14:02:38.870 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:38.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.983+0000 7f1bdd97a640 1 Processor -- start 2024-04-25T14:02:38.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.983+0000 7f1bdd97a640 1 -- start start 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bdd97a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd80b67b0 0x7f1bd80b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bdd97a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1bd8164070 0x7f1bd8164480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bdd97a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bd81649c0 con 0x7f1bd80d1db0 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bdd97a640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1bd80b9860 con 0x7f1bd8164070 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bdd97a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd80b92a0 con 0x7f1bd80b67b0 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bce7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1bd8164070 0x7f1bd8164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bce7fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1bd8164070 0x7f1bd8164480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56362/0 (socket says 172.21.15.150:56362) 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bce7fc640 1 -- 172.21.15.150:0/4142564340 learned_addr learned my addr 172.21.15.150:0/4142564340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.984+0000 7f1bd6ffd640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd80b67b0 0x7f1bd80b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bce7fc640 1 -- 172.21.15.150:0/4142564340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd80b67b0 msgr2=0x7f1bd80b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:38.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bce7fc640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd80b67b0 0x7f1bd80b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bce7fc640 1 -- 172.21.15.150:0/4142564340 >> v1:172.21.15.187:6789/0 conn(0x7f1bd80d1db0 legacy=0x7f1bd80d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bce7fc640 1 -- 172.21.15.150:0/4142564340 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd80b9580 con 0x7f1bd8164070 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bce7fc640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd8164070 0x7f1bd8164480 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009870 tx=0x7f1bc002a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd03f5f0aa034055 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.985+0000 7f1bd67fc640 1 -- 172.21.15.150:0/4142564340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0037070 con 0x7f1bd8164070 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bd67fc640 1 -- 172.21.15.150:0/4142564340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc0032440 con 0x7f1bd8164070 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bd67fc640 1 -- 172.21.15.150:0/4142564340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0032810 con 0x7f1bd8164070 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 -- 172.21.15.150:0/4142564340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd8164070 msgr2=0x7f1bd8164480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd8164070 0x7f1bd8164480 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009870 tx=0x7f1bc002a1c0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 -- 172.21.15.150:0/4142564340 shutdown_connections 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd8164070 0x7f1bd8164480 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/4142564340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd80b67b0 0x7f1bd80b6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 -- 172.21.15.150:0/4142564340 >> 172.21.15.150:0/4142564340 conn(0x7f1bd80d3960 msgr2=0x7f1bd80d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 -- 172.21.15.150:0/4142564340 shutdown_connections 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.986+0000 7f1bdd97a640 1 -- 172.21.15.150:0/4142564340 wait complete. 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.988+0000 7f1bdd97a640 1 Processor -- start 2024-04-25T14:02:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.988+0000 7f1bdd97a640 1 -- start start 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bd80b67b0 0x7f1bd8157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd8164070 0x7f1bd8162600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd80b8e60 con 0x7f1bd80d1db0 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd80d18b0 con 0x7f1bd80b67b0 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.989+0000 7f1bdd97a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1bd80b8fe0 con 0x7f1bd8164070 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56376/0 (socket says 172.21.15.150:56376) 2024-04-25T14:02:38.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 -- 172.21.15.150:0/1085070538 learned_addr learned my addr 172.21.15.150:0/1085070538 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bd6ffd640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bd80b67b0 0x7f1bd8157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bd77fe640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd8164070 0x7f1bd8162600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 -- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd8164070 msgr2=0x7f1bd8162600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd8164070 0x7f1bd8162600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 -- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bd80b67b0 msgr2=0x7f1bd8157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bd80b67b0 0x7f1bd8157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 -- 172.21.15.150:0/1085070538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd806ea30 con 0x7f1bd80d1db0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bce7fc640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0037040 con 0x7f1bd80d1db0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bc0042070 con 0x7f1bd80d1db0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd806ed10 con 0x7f1bd80d1db0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0030880 con 0x7f1bd80d1db0 2024-04-25T14:02:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.990+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd806f2d0 con 0x7f1bd80d1db0 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.991+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd8076a40 con 0x7f1bd80d1db0 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.992+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1bc002ad10 con 0x7f1bd80d1db0 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.993+0000 7f1bd4ff9640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 0x7f1bb80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.993+0000 7f1bd6ffd640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 0x7f1bb80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.993+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f1bc00b59c0 con 0x7f1bd80d1db0 2024-04-25T14:02:38.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.993+0000 7f1bd6ffd640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 0x7f1bb80769f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bd808ca70 tx=0x7f1bc8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:38.997+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1bc00b7e20 con 0x7f1bd80d1db0 2024-04-25T14:02:39.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.170+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1bd815d680 con 0x7f1bd80d1db0 2024-04-25T14:02:39.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.170+0000 7f1bd4ff9640 1 -- 172.21.15.150:0/1085070538 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f1bc007f850 con 0x7f1bd80d1db0 2024-04-25T14:02:39.169 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:39.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 msgr2=0x7f1bb80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 0x7f1bb80769f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1bd808ca70 tx=0x7f1bc8009190 comp rx=0 tx=0).stop 2024-04-25T14:02:39.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 msgr2=0x7f1bd815cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0000c00 tx=0x7f1bc00046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 shutdown_connections 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1bb80744e0 0x7f1bb80769f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1bd8164070 0x7f1bd8162600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1bd80d1db0 0x7f1bd815cea0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 --2- 172.21.15.150:0/1085070538 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1bd80b67b0 0x7f1bd8157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 >> 172.21.15.150:0/1085070538 conn(0x7f1bd80d3960 msgr2=0x7f1bd80b3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 shutdown_connections 2024-04-25T14:02:39.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.172+0000 7f1bdd97a640 1 -- 172.21.15.150:0/1085070538 wait complete. 2024-04-25T14:02:39.182 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.292+0000 7f10b1c67640 1 Processor -- start 2024-04-25T14:02:39.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.292+0000 7f10b1c67640 1 -- start start 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10b1c67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10ac1601d0 0x7f10ac0d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10b1c67640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac06c570 con 0x7f10ac06da00 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10b1c67640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10ac1605e0 con 0x7f10ac0b9070 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10b1c67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f10ac0d5980 con 0x7f10ac1601d0 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10ab7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10ac1601d0 0x7f10ac0d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10abfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f10ac06da00 0x7f10ac163ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41232/0 (socket says 172.21.15.150:41232) 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.293+0000 7f10abfff640 1 -- 172.21.15.150:0/1705732090 learned_addr learned my addr 172.21.15.150:0/1705732090 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10ab7fe640 1 -- 172.21.15.150:0/1705732090 >> v1:172.21.15.187:6789/0 conn(0x7f10ac0b9070 legacy=0x7f10ac06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10ab7fe640 1 -- 172.21.15.150:0/1705732090 >> v1:172.21.15.150:6789/0 conn(0x7f10ac06da00 legacy=0x7f10ac163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10ab7fe640 1 -- 172.21.15.150:0/1705732090 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac06c810 con 0x7f10ac1601d0 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10ab7fe640 1 --2- 172.21.15.150:0/1705732090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1601d0 0x7f10ac0d5100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f10a00097a0 tx=0x7f10a002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f07a7e3c328b9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10aa7fc640 1 -- 172.21.15.150:0/1705732090 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0037070 con 0x7f10ac1601d0 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10aa7fc640 1 -- 172.21.15.150:0/1705732090 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a0032440 con 0x7f10ac1601d0 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.294+0000 7f10aa7fc640 1 -- 172.21.15.150:0/1705732090 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0032810 con 0x7f10ac1601d0 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 -- 172.21.15.150:0/1705732090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1601d0 msgr2=0x7f10ac0d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 --2- 172.21.15.150:0/1705732090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1601d0 0x7f10ac0d5100 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f10a00097a0 tx=0x7f10a002a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 -- 172.21.15.150:0/1705732090 shutdown_connections 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 --2- 172.21.15.150:0/1705732090 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac1601d0 0x7f10ac0d5100 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 -- 172.21.15.150:0/1705732090 >> 172.21.15.150:0/1705732090 conn(0x7f10ac0d5d90 msgr2=0x7f10ac0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 -- 172.21.15.150:0/1705732090 shutdown_connections 2024-04-25T14:02:39.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.295+0000 7f10b1c67640 1 -- 172.21.15.150:0/1705732090 wait complete. 2024-04-25T14:02:39.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.297+0000 7f10b1c67640 1 Processor -- start 2024-04-25T14:02:39.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.297+0000 7f10b1c67640 1 -- start start 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10ac0b9070 0x7f10ac0c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10ac1601d0 0x7f10ac15a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac0d5640 con 0x7f10ac06da00 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10ac06cdd0 con 0x7f10ac1601d0 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10b1c67640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10ac06de10 con 0x7f10ac0b9070 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10ab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10ab7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56384/0 (socket says 172.21.15.150:56384) 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10ab7fe640 1 -- 172.21.15.150:0/233598078 learned_addr learned my addr 172.21.15.150:0/233598078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10aaffd640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10ac0b9070 0x7f10ac0c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.298+0000 7f10abfff640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10ac1601d0 0x7f10ac15a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 -- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10ac0b9070 msgr2=0x7f10ac0c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10ac0b9070 0x7f10ac0c98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 -- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10ac1601d0 msgr2=0x7f10ac15a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10ac1601d0 0x7f10ac15a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 -- 172.21.15.150:0/233598078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ac0c9e00 con 0x7f10ac06da00 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10ab7fe640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a00098c0 tx=0x7f10a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0037040 con 0x7f10ac06da00 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10a0042070 con 0x7f10ac06da00 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ac0ca310 con 0x7f10ac06da00 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a0030a70 con 0x7f10ac06da00 2024-04-25T14:02:39.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.299+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ac0ca8d0 con 0x7f10ac06da00 2024-04-25T14:02:39.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.300+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ac076a40 con 0x7f10ac06da00 2024-04-25T14:02:39.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.301+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f10a0030c10 con 0x7f10ac06da00 2024-04-25T14:02:39.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.301+0000 7f1093fff640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 0x7f10980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.301+0000 7f10aaffd640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 0x7f10980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.301+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f10a00b5e20 con 0x7f10ac06da00 2024-04-25T14:02:39.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.302+0000 7f10aaffd640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 0x7f10980769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0d2690 tx=0x7f109c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.306+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f10a004c020 con 0x7f10ac06da00 2024-04-25T14:02:39.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.520+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10ac0c9660 con 0x7f10ac06da00 2024-04-25T14:02:39.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.520+0000 7f1093fff640 1 -- 172.21.15.150:0/233598078 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f10a00882f0 con 0x7f10ac06da00 2024-04-25T14:02:39.520 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:39.520 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:39.521 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 msgr2=0x7f10980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 0x7f10980769c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0d2690 tx=0x7f109c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 msgr2=0x7f10ac159720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f10a00098c0 tx=0x7f10a00046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 shutdown_connections 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10980744b0 0x7f10980769c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10ac1601d0 0x7f10ac15a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10ac0b9070 0x7f10ac0c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 --2- 172.21.15.150:0/233598078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10ac06da00 0x7f10ac159720 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 >> 172.21.15.150:0/233598078 conn(0x7f10ac0d5d90 msgr2=0x7f10ac15e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 shutdown_connections 2024-04-25T14:02:39.524 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.522+0000 7f10b1c67640 1 -- 172.21.15.150:0/233598078 wait complete. 2024-04-25T14:02:39.533 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:39.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.642+0000 7f031ddce640 1 Processor -- start 2024-04-25T14:02:39.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.642+0000 7f031ddce640 1 -- start start 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f031ddce640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03180d4200 0x7f03180d0b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f031ddce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03180d10c0 0x7f03180b9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f031ddce640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03180d14d0 con 0x7f03180b4e00 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f031ddce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f03180b9680 con 0x7f03180d10c0 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f031ddce640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f031806d790 con 0x7f03180d4200 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f03177fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f03180b4e00 0x7f031806cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41242/0 (socket says 172.21.15.150:41242) 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f0317fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f03180d10c0 0x7f03180b9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f03177fe640 1 -- 172.21.15.150:0/4251541379 learned_addr learned my addr 172.21.15.150:0/4251541379 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.643+0000 7f0316ffd640 1 --2- 172.21.15.150:0/4251541379 >> v2:172.21.15.187:3300/0 conn(0x7f03180d4200 0x7f03180d0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f0317fff640 1 -- 172.21.15.150:0/4251541379 >> v2:172.21.15.187:3300/0 conn(0x7f03180d4200 msgr2=0x7f03180d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f0317fff640 1 --2- 172.21.15.150:0/4251541379 >> v2:172.21.15.187:3300/0 conn(0x7f03180d4200 0x7f03180d0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f0317fff640 1 -- 172.21.15.150:0/4251541379 >> v1:172.21.15.150:6789/0 conn(0x7f03180b4e00 legacy=0x7f031806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f0317fff640 1 -- 172.21.15.150:0/4251541379 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031806da70 con 0x7f03180d10c0 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f0317fff640 1 --2- 172.21.15.150:0/4251541379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03180b9b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03180d3de0 tx=0x7f031002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b6853362ef4611 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.644+0000 7f03167fc640 1 -- 172.21.15.150:0/4251541379 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310007740 con 0x7f03180d10c0 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f03167fc640 1 -- 172.21.15.150:0/4251541379 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03100078e0 con 0x7f03180d10c0 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f03167fc640 1 -- 172.21.15.150:0/4251541379 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0310007cb0 con 0x7f03180d10c0 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 -- 172.21.15.150:0/4251541379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 msgr2=0x7f03180b9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 --2- 172.21.15.150:0/4251541379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03180b9b10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03180d3de0 tx=0x7f031002a610 comp rx=0 tx=0).stop 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 -- 172.21.15.150:0/4251541379 shutdown_connections 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 --2- 172.21.15.150:0/4251541379 >> v2:172.21.15.187:3300/0 conn(0x7f03180d4200 0x7f03180d0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 --2- 172.21.15.150:0/4251541379 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03180b9b10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 -- 172.21.15.150:0/4251541379 >> 172.21.15.150:0/4251541379 conn(0x7f03180d63d0 msgr2=0x7f03180d2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 -- 172.21.15.150:0/4251541379 shutdown_connections 2024-04-25T14:02:39.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.645+0000 7f031ddce640 1 -- 172.21.15.150:0/4251541379 wait complete. 2024-04-25T14:02:39.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.647+0000 7f031ddce640 1 Processor -- start 2024-04-25T14:02:39.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.647+0000 7f031ddce640 1 -- start start 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.648+0000 7f031ddce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03180b4e00 0x7f0318155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03180d4200 0x7f03180ce250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03180ba050 con 0x7f03180d10c0 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f03180b92a0 con 0x7f03180d4200 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f03180ba1d0 con 0x7f03180b4e00 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56392/0 (socket says 172.21.15.150:56392) 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 -- 172.21.15.150:0/2610424482 learned_addr learned my addr 172.21.15.150:0/2610424482 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:39.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0317fff640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03180d4200 0x7f03180ce250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f03177fe640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03180b4e00 0x7f0318155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 -- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03180b4e00 msgr2=0x7f0318155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03180b4e00 0x7f0318155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 -- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03180d4200 msgr2=0x7f03180ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03180d4200 0x7f03180ce250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 -- 172.21.15.150:0/2610424482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0318161490 con 0x7f03180d10c0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f0316ffd640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f030800e8a0 tx=0x7f030800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0308017070 con 0x7f03180d10c0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030800c940 con 0x7f03180d10c0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0318161af0 con 0x7f03180d10c0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800cd30 con 0x7f03180d10c0 2024-04-25T14:02:39.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.649+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03181620b0 con 0x7f03180d10c0 2024-04-25T14:02:39.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.651+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03181611e0 con 0x7f03180d10c0 2024-04-25T14:02:39.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.651+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f030801a070 con 0x7f03180d10c0 2024-04-25T14:02:39.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.652+0000 7f02fffff640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 0x7f03040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.652+0000 7f03177fe640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 0x7f03040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.652+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f0308095790 con 0x7f03180d10c0 2024-04-25T14:02:39.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.652+0000 7f03177fe640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 0x7f03040769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f030c000f60 tx=0x7f030c007e20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.656+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f030805f590 con 0x7f03180d10c0 2024-04-25T14:02:39.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.834+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0318076a40 con 0x7f03180d10c0 2024-04-25T14:02:39.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.835+0000 7f02fffff640 1 -- 172.21.15.150:0/2610424482 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f0308063890 con 0x7f03180d10c0 2024-04-25T14:02:39.835 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:39.835 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 msgr2=0x7f03040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 0x7f03040769c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f030c000f60 tx=0x7f030c007e20 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 msgr2=0x7f03181608b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f030800e8a0 tx=0x7f030800ed60 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 shutdown_connections 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03040744b0 0x7f03040769c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03180d4200 0x7f03180ce250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03180d10c0 0x7f03181608b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 --2- 172.21.15.150:0/2610424482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03180b4e00 0x7f0318155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 >> 172.21.15.150:0/2610424482 conn(0x7f03180d63d0 msgr2=0x7f03180684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.837+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 shutdown_connections 2024-04-25T14:02:39.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.838+0000 7f031ddce640 1 -- 172.21.15.150:0/2610424482 wait complete. 2024-04-25T14:02:39.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:39.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:39.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.960+0000 7f5f6726e640 1 Processor -- start 2024-04-25T14:02:39.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.960+0000 7f5f6726e640 1 -- start start 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f6726e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f600b5ad0 0x7f5f6006ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f6726e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5f60163d80 0x7f5f600d1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f6726e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f6006ff50 con 0x7f5f60160460 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f6726e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6006f3d0 con 0x7f5f600b5ad0 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f6726e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5f6006f6b0 con 0x7f5f60163d80 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f64809640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f600b5ad0 0x7f5f6006ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f64809640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5f600b5ad0 0x7f5f6006ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56394/0 (socket says 172.21.15.150:56394) 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.961+0000 7f5f64809640 1 -- 172.21.15.150:0/2471712827 learned_addr learned my addr 172.21.15.150:0/2471712827 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f6580b640 1 --2- 172.21.15.150:0/2471712827 >> v2:172.21.15.187:3300/0 conn(0x7f5f60163d80 0x7f5f600d1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f64809640 1 -- 172.21.15.150:0/2471712827 >> v2:172.21.15.187:3300/0 conn(0x7f5f60163d80 msgr2=0x7f5f600d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:39.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f64809640 1 --2- 172.21.15.150:0/2471712827 >> v2:172.21.15.187:3300/0 conn(0x7f5f60163d80 0x7f5f600d1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f64809640 1 -- 172.21.15.150:0/2471712827 >> v1:172.21.15.187:6789/0 conn(0x7f5f60160460 legacy=0x7f5f600b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.962 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f64809640 1 -- 172.21.15.150:0/2471712827 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f6006f990 con 0x7f5f600b5ad0 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.962+0000 7f5f64809640 1 --2- 172.21.15.150:0/2471712827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b5ad0 0x7f5f6006ed40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f600b2b40 tx=0x7f5f5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee1eb81f8d1d1dfe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f57fff640 1 -- 172.21.15.150:0/2471712827 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50037070 con 0x7f5f600b5ad0 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f57fff640 1 -- 172.21.15.150:0/2471712827 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f50032440 con 0x7f5f600b5ad0 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f57fff640 1 -- 172.21.15.150:0/2471712827 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50032810 con 0x7f5f600b5ad0 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 -- 172.21.15.150:0/2471712827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b5ad0 msgr2=0x7f5f6006ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 --2- 172.21.15.150:0/2471712827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b5ad0 0x7f5f6006ed40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f5f600b2b40 tx=0x7f5f5002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 -- 172.21.15.150:0/2471712827 shutdown_connections 2024-04-25T14:02:39.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 --2- 172.21.15.150:0/2471712827 >> v2:172.21.15.187:3300/0 conn(0x7f5f60163d80 0x7f5f600d1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 --2- 172.21.15.150:0/2471712827 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b5ad0 0x7f5f6006ed40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 -- 172.21.15.150:0/2471712827 >> 172.21.15.150:0/2471712827 conn(0x7f5f600d03c0 msgr2=0x7f5f600d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 -- 172.21.15.150:0/2471712827 shutdown_connections 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.963+0000 7f5f6726e640 1 -- 172.21.15.150:0/2471712827 wait complete. 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.965+0000 7f5f6726e640 1 Processor -- start 2024-04-25T14:02:39.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.965+0000 7f5f6726e640 1 -- start start 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f60163d80 0x7f5f6015aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f60163120 0x7f5f600b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5f600ce5c0 con 0x7f5f600b0860 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5f600b3030 con 0x7f5f60163d80 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.966+0000 7f5f6726e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5f600d1da0 con 0x7f5f60163120 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56410/0 (socket says 172.21.15.150:56410) 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 -- 172.21.15.150:0/1090322084 learned_addr learned my addr 172.21.15.150:0/1090322084 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6500a640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f60163d80 0x7f5f6015aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f64809640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f60163120 0x7f5f600b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 -- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f60163120 msgr2=0x7f5f600b0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f60163120 0x7f5f600b0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 -- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f60163d80 msgr2=0x7f5f6015aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f60163d80 0x7f5f6015aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 -- 172.21.15.150:0/1090322084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f600b11b0 con 0x7f5f600b0860 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6580b640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d30 tx=0x7f5f5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58018020 con 0x7f5f600b0860 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f58014480 con 0x7f5f600b0860 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f58014870 con 0x7f5f600b0860 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f600aeb00 con 0x7f5f600b0860 2024-04-25T14:02:39.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.967+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f60161ad0 con 0x7f5f600b0860 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.969+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f6015afb0 con 0x7f5f600b0860 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.969+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5f58005d10 con 0x7f5f600b0860 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.970+0000 7f5f55ffb640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 0x7f5f440769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.970+0000 7f5f6500a640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 0x7f5f440769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.970+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f5f580961b0 con 0x7f5f600b0860 2024-04-25T14:02:39.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.970+0000 7f5f6500a640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 0x7f5f440769f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5f6006e7e0 tx=0x7f5f5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:39.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:39.975+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5f58022030 con 0x7f5f600b0860 2024-04-25T14:02:40.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.148+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5f60076a40 con 0x7f5f600b0860 2024-04-25T14:02:40.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.149+0000 7f5f55ffb640 1 -- 172.21.15.150:0/1090322084 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) ==== 93+0+10 (secure 0 0 0) 0x7f5f58010040 con 0x7f5f600b0860 2024-04-25T14:02:40.148 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:40.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 msgr2=0x7f5f440769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 0x7f5f440769f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5f6006e7e0 tx=0x7f5f5c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 msgr2=0x7f5f600b0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009d30 tx=0x7f5f5800c490 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 shutdown_connections 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5f440744e0 0x7f5f440769f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5f600b0860 0x7f5f600b0c70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5f60163120 0x7f5f600b0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 --2- 172.21.15.150:0/1090322084 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5f60163d80 0x7f5f6015aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 >> 172.21.15.150:0/1090322084 conn(0x7f5f600d03c0 msgr2=0x7f5f600ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 shutdown_connections 2024-04-25T14:02:40.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.151+0000 7f5f6726e640 1 -- 172.21.15.150:0/1090322084 wait complete. 2024-04-25T14:02:40.161 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:40.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.273+0000 7fad14092640 1 Processor -- start 2024-04-25T14:02:40.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.273+0000 7fad14092640 1 -- start start 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c1576d0 0x7fad0c0d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad0c0b36e0 0x7fad0c15db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fad0c09b4a0 0x7fad0c09b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c0b3250 con 0x7fad0c0b36e0 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fad0c15e3d0 con 0x7fad0c09b4a0 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad14092640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c09bdf0 con 0x7fad0c1576d0 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad0c0b36e0 0x7fad0c15db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fad0c0b36e0 0x7fad0c15db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56422/0 (socket says 172.21.15.150:56422) 2024-04-25T14:02:40.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 -- 172.21.15.150:0/1168537712 learned_addr learned my addr 172.21.15.150:0/1168537712 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1262f640 1 --2- 172.21.15.150:0/1168537712 >> v2:172.21.15.187:3300/0 conn(0x7fad0c09b4a0 0x7fad0c09b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad11e2e640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c1576d0 0x7fad0c0d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 -- 172.21.15.150:0/1168537712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c1576d0 msgr2=0x7fad0c0d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c1576d0 0x7fad0c0d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 -- 172.21.15.150:0/1168537712 >> v2:172.21.15.187:3300/0 conn(0x7fad0c09b4a0 msgr2=0x7fad0c09b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 --2- 172.21.15.150:0/1168537712 >> v2:172.21.15.187:3300/0 conn(0x7fad0c09b4a0 0x7fad0c09b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.275+0000 7fad1162d640 1 -- 172.21.15.150:0/1168537712 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c0b3af0 con 0x7fad0c0b36e0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.276+0000 7fad1162d640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c0b36e0 0x7fad0c15db80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fad0c157ae0 tx=0x7facfc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe2b5d6da975029 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.276+0000 7fad10e2c640 1 -- 172.21.15.150:0/1168537712 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc037070 con 0x7fad0c0b36e0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.276+0000 7fad10e2c640 1 -- 172.21.15.150:0/1168537712 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facfc032440 con 0x7fad0c0b36e0 2024-04-25T14:02:40.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.276+0000 7fad10e2c640 1 -- 172.21.15.150:0/1168537712 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facfc032810 con 0x7fad0c0b36e0 2024-04-25T14:02:40.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 -- 172.21.15.150:0/1168537712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c0b36e0 msgr2=0x7fad0c15db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c0b36e0 0x7fad0c15db80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fad0c157ae0 tx=0x7facfc02a140 comp rx=0 tx=0).stop 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 -- 172.21.15.150:0/1168537712 shutdown_connections 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 --2- 172.21.15.150:0/1168537712 >> v2:172.21.15.187:3300/0 conn(0x7fad0c09b4a0 0x7fad0c09b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c0b36e0 0x7fad0c15db80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 --2- 172.21.15.150:0/1168537712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c1576d0 0x7fad0c0d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 -- 172.21.15.150:0/1168537712 >> 172.21.15.150:0/1168537712 conn(0x7fad0c0b76d0 msgr2=0x7fad0c0b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 -- 172.21.15.150:0/1168537712 shutdown_connections 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.277+0000 7fad14092640 1 -- 172.21.15.150:0/1168537712 wait complete. 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.278+0000 7fad14092640 1 Processor -- start 2024-04-25T14:02:40.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.279+0000 7fad14092640 1 -- start start 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 0x7fad0c0d1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 0x7fad0c1564e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b2ee0 con 0x7fad0c1576d0 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fad0c0b2c00 con 0x7fad0c0b36e0 2024-04-25T14:02:40.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad14092640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fad0c15e0c0 con 0x7fad0c09b4a0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56424/0 (socket says 172.21.15.150:56424) 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 -- 172.21.15.150:0/2365259508 learned_addr learned my addr 172.21.15.150:0/2365259508 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad11e2e640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 0x7fad0c0d1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1162d640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 0x7fad0c1564e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 -- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 msgr2=0x7fad0c0d1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 0x7fad0c0d1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 -- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 msgr2=0x7fad0c1564e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 0x7fad0c1564e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 -- 172.21.15.150:0/2365259508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad0c0d06d0 con 0x7fad0c1576d0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad11e2e640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 0x7fad0c0d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1162d640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 0x7fad0c1564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad1262f640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad0400ef00 tx=0x7fad0400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.280+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04018020 con 0x7fad0c1576d0 2024-04-25T14:02:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.281+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad0400bd30 con 0x7fad0c1576d0 2024-04-25T14:02:40.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.281+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad0c0d09b0 con 0x7fad0c1576d0 2024-04-25T14:02:40.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.281+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad04014640 con 0x7fad0c1576d0 2024-04-25T14:02:40.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.281+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad0c0d0f70 con 0x7fad0c1576d0 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.281+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad0c076a40 con 0x7fad0c1576d0 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.282+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fad0401b020 con 0x7fad0c1576d0 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.283+0000 7fad027fc640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 0x7facf00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.283+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fad04095e90 con 0x7fad0c1576d0 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.283+0000 7fad11e2e640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 0x7facf00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.283+0000 7fad11e2e640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 0x7facf00769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0741c0 tx=0x7fad08008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.287+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad04022020 con 0x7fad0c1576d0 2024-04-25T14:02:40.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.460+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad0c06f8a0 con 0x7fad0c1576d0 2024-04-25T14:02:40.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.460+0000 7fad027fc640 1 -- 172.21.15.150:0/2365259508 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) ==== 104+0+11 (secure 0 0 0) 0x7fad04066020 con 0x7fad0c1576d0 2024-04-25T14:02:40.460 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 msgr2=0x7facf00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 0x7facf00769c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fad0c0741c0 tx=0x7fad08008040 comp rx=0 tx=0).stop 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 msgr2=0x7fad0c14b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fad0400ef00 tx=0x7fad0400b480 comp rx=0 tx=0).stop 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 shutdown_connections 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7facf00744b0 0x7facf00769c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fad0c1576d0 0x7fad0c14b0e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fad0c0b36e0 0x7fad0c1564e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 --2- 172.21.15.150:0/2365259508 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fad0c09b4a0 0x7fad0c0d1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 >> 172.21.15.150:0/2365259508 conn(0x7fad0c0b76d0 msgr2=0x7fad0c152cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 shutdown_connections 2024-04-25T14:02:40.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.462+0000 7fad14092640 1 -- 172.21.15.150:0/2365259508 wait complete. 2024-04-25T14:02:40.473 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:40.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.584+0000 7f9c42ad9640 1 Processor -- start 2024-04-25T14:02:40.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.584+0000 7f9c42ad9640 1 -- start start 2024-04-25T14:02:40.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c42ad9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 0x7f9c3c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c42ad9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c06ff50 con 0x7f9c3c0b5ad0 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c42ad9640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c3c06f3d0 con 0x7f9c3c160460 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c42ad9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c06f6b0 con 0x7f9c3c163d80 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9c3c0b5ad0 0x7f9c3c0d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41252/0 (socket says 172.21.15.150:41252) 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3bfff640 1 -- 172.21.15.150:0/1559949234 learned_addr learned my addr 172.21.15.150:0/1559949234 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c41076640 1 --2- 172.21.15.150:0/1559949234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 0x7f9c3c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2667243901 0 0) 0x7f9c3c06ff50 con 0x7f9c3c0b5ad0 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c24003520 con 0x7f9c3c0b5ad0 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 484378400 0 0) 0x7f9c3c06f3d0 con 0x7f9c3c160460 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c3c06ff50 con 0x7f9c3c160460 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4174725345 0 0) 0x7f9c24003520 con 0x7f9c3c0b5ad0 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c41076640 1 -- 172.21.15.150:0/1559949234 >> v1:172.21.15.187:6789/0 conn(0x7f9c3c160460 legacy=0x7f9c3c0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c41076640 1 -- 172.21.15.150:0/1559949234 >> v1:172.21.15.150:6789/0 conn(0x7f9c3c0b5ad0 legacy=0x7f9c3c0d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.586+0000 7f9c41076640 1 -- 172.21.15.150:0/1559949234 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c06f990 con 0x7f9c3c163d80 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.587+0000 7f9c41076640 1 --2- 172.21.15.150:0/1559949234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 0x7f9c3c06ed40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009dc0 tx=0x7f9c3402a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da268e4ac9f8f2f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.587+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c30001670 con 0x7f9c3c163d80 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.587+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c34007540 con 0x7f9c3c163d80 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.587+0000 7f9c3b7fe640 1 -- 172.21.15.150:0/1559949234 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c34007910 con 0x7f9c3c163d80 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 -- 172.21.15.150:0/1559949234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 msgr2=0x7f9c3c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/1559949234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 0x7f9c3c06ed40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9c34009dc0 tx=0x7f9c3402a5d0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 -- 172.21.15.150:0/1559949234 shutdown_connections 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/1559949234 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c163d80 0x7f9c3c06ed40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 -- 172.21.15.150:0/1559949234 >> 172.21.15.150:0/1559949234 conn(0x7f9c3c0d03c0 msgr2=0x7f9c3c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 -- 172.21.15.150:0/1559949234 shutdown_connections 2024-04-25T14:02:40.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.588+0000 7f9c42ad9640 1 -- 172.21.15.150:0/1559949234 wait complete. 2024-04-25T14:02:40.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.590+0000 7f9c42ad9640 1 Processor -- start 2024-04-25T14:02:40.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.590+0000 7f9c42ad9640 1 -- start start 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 0x7f9c3c0b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c1f9fe0 0x7f9c3c1592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 0x7f9c3c1f97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c0ce5c0 con 0x7f9c3c1f9390 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9c3c0b3030 con 0x7f9c3c163d80 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c42ad9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c3c0d1f70 con 0x7f9c3c1f9fe0 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c41076640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 0x7f9c3c1f97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c41076640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 0x7f9c3c1f97a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56434/0 (socket says 172.21.15.150:56434) 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c40875640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 0x7f9c3c0b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c41076640 1 -- 172.21.15.150:0/494748994 learned_addr learned my addr 172.21.15.150:0/494748994 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:40.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.591+0000 7f9c3bfff640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c1f9fe0 0x7f9c3c1592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 -- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c1f9fe0 msgr2=0x7f9c3c1592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c1f9fe0 0x7f9c3c1592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 -- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 msgr2=0x7f9c3c1f97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 0x7f9c3c1f97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 -- 172.21.15.150:0/494748994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c3c1f9ce0 con 0x7f9c3c163d80 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c40875640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 0x7f9c3c0b89e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c3c0 tx=0x7f9c2c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c017070 con 0x7f9c3c163d80 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2c0052f0 con 0x7f9c3c163d80 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c003830 con 0x7f9c3c163d80 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c3c161830 con 0x7f9c3c163d80 2024-04-25T14:02:40.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.592+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c3c0b68e0 con 0x7f9c3c163d80 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.593+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c3c076a40 con 0x7f9c3c163d80 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.594+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9c2c01a070 con 0x7f9c3c163d80 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.594+0000 7f9c397fa640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 0x7f9c280769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.594+0000 7f9c3bfff640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 0x7f9c280769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.594+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f9c2c0946d0 con 0x7f9c3c163d80 2024-04-25T14:02:40.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.595+0000 7f9c3bfff640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 0x7f9c280769c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c300011e0 tx=0x7f9c300016f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.599+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c3c1f9ce0 con 0x7f9c3c163d80 2024-04-25T14:02:40.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.774+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9c3c0b98c0 con 0x7f9c3c163d80 2024-04-25T14:02:40.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.775+0000 7f9c397fa640 1 -- 172.21.15.150:0/494748994 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) ==== 100+0+11 (secure 0 0 0) 0x7f9c2c05e540 con 0x7f9c3c163d80 2024-04-25T14:02:40.774 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:40.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 msgr2=0x7f9c280769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 0x7f9c280769c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c300011e0 tx=0x7f9c300016f0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 msgr2=0x7f9c3c0b89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 0x7f9c3c0b89e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c00c3c0 tx=0x7f9c2c00c880 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 shutdown_connections 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c280744b0 0x7f9c280769c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c3c1f9390 0x7f9c3c1f97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c3c1f9fe0 0x7f9c3c1592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 --2- 172.21.15.150:0/494748994 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c3c163d80 0x7f9c3c0b89e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 >> 172.21.15.150:0/494748994 conn(0x7f9c3c0d03c0 msgr2=0x7f9c3c15ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 shutdown_connections 2024-04-25T14:02:40.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.777+0000 7f9c42ad9640 1 -- 172.21.15.150:0/494748994 wait complete. 2024-04-25T14:02:40.788 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:40.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.899+0000 7f101e069640 1 Processor -- start 2024-04-25T14:02:40.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.899+0000 7f101e069640 1 -- start start 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f101e069640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10180cf8e0 0x7f10181605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f101e069640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cdfe0 0x7f10180d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f101e069640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10180d3570 con 0x7f10180b3a00 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f101e069640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f10180b4800 con 0x7f10180cf8e0 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f101e069640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10180b4ae0 con 0x7f10180cdfe0 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f10177fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10180cf8e0 0x7f10181605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f10177fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f10180cf8e0 0x7f10181605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56442/0 (socket says 172.21.15.150:56442) 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f10177fe640 1 -- 172.21.15.150:0/1477670612 learned_addr learned my addr 172.21.15.150:0/1477670612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.900+0000 7f1017fff640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cdfe0 0x7f10180d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10177fe640 1 -- 172.21.15.150:0/1477670612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cdfe0 msgr2=0x7f10180d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:40.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10177fe640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cdfe0 0x7f10180d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10177fe640 1 -- 172.21.15.150:0/1477670612 >> v1:172.21.15.187:6789/0 conn(0x7f10180b3a00 legacy=0x7f10180b3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:40.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10177fe640 1 -- 172.21.15.150:0/1477670612 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10180b4dc0 con 0x7f10180cf8e0 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10177fe640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180cf8e0 0x7f10181605d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1008009870 tx=0x7f100802a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9546586c25c3951 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10167fc640 1 -- 172.21.15.150:0/1477670612 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008037070 con 0x7f10180cf8e0 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.901+0000 7f10167fc640 1 -- 172.21.15.150:0/1477670612 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1008032440 con 0x7f10180cf8e0 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f10167fc640 1 -- 172.21.15.150:0/1477670612 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008032810 con 0x7f10180cf8e0 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 -- 172.21.15.150:0/1477670612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180cf8e0 msgr2=0x7f10181605d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180cf8e0 0x7f10181605d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1008009870 tx=0x7f100802a210 comp rx=0 tx=0).stop 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 -- 172.21.15.150:0/1477670612 shutdown_connections 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cdfe0 0x7f10180d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 --2- 172.21.15.150:0/1477670612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180cf8e0 0x7f10181605d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 -- 172.21.15.150:0/1477670612 >> 172.21.15.150:0/1477670612 conn(0x7f101809b4a0 msgr2=0x7f101809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:40.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 -- 172.21.15.150:0/1477670612 shutdown_connections 2024-04-25T14:02:40.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.902+0000 7f101e069640 1 -- 172.21.15.150:0/1477670612 wait complete. 2024-04-25T14:02:40.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.904+0000 7f101e069640 1 Processor -- start 2024-04-25T14:02:40.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.904+0000 7f101e069640 1 -- start start 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10180cdfe0 0x7f1018163080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf8e0 0x7f10181f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f10180d2360 con 0x7f10180b3a00 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f10180d21e0 con 0x7f10180cdfe0 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.905+0000 7f101e069640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f10180d24e0 con 0x7f10180cf8e0 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56444/0 (socket says 172.21.15.150:56444) 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 -- 172.21.15.150:0/788515821 learned_addr learned my addr 172.21.15.150:0/788515821 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f1016ffd640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10180cdfe0 0x7f1018163080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f1017fff640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf8e0 0x7f10181f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 -- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf8e0 msgr2=0x7f10181f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf8e0 0x7f10181f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 -- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10180cdfe0 msgr2=0x7f1018163080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10180cdfe0 0x7f1018163080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 -- 172.21.15.150:0/788515821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1018159bf0 con 0x7f10180b3a00 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f10177fe640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f10080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008041070 con 0x7f10180b3a00 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f100803c870 con 0x7f10180b3a00 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1008037040 con 0x7f10180b3a00 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101815cba0 con 0x7f10180b3a00 2024-04-25T14:02:40.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.906+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f101815d160 con 0x7f10180b3a00 2024-04-25T14:02:40.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.907+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1018076a40 con 0x7f10180b3a00 2024-04-25T14:02:40.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.908+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1008044070 con 0x7f10180b3a00 2024-04-25T14:02:40.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.909+0000 7f0ffffff640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 0x7f10040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:40.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.909+0000 7f1016ffd640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 0x7f10040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:40.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.909+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f10080b5f80 con 0x7f10180b3a00 2024-04-25T14:02:40.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.909+0000 7f1016ffd640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 0x7f10040769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10180cf220 tx=0x7f100c009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:40.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:40.913+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f100807fd80 con 0x7f10180b3a00 2024-04-25T14:02:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.089+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f1018071f30 con 0x7f10180b3a00 2024-04-25T14:02:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.090+0000 7f0ffffff640 1 -- 172.21.15.150:0/788515821 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v24) ==== 103+0+11 (secure 0 0 0) 0x7f1008084080 con 0x7f10180b3a00 2024-04-25T14:02:41.089 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 msgr2=0x7f10040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 0x7f10040769c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10180cf220 tx=0x7f100c009060 comp rx=0 tx=0).stop 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 msgr2=0x7f1018159110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f10080046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 shutdown_connections 2024-04-25T14:02:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f10040744b0 0x7f10040769c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f10180cf8e0 0x7f10181f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f10180cdfe0 0x7f1018163080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 --2- 172.21.15.150:0/788515821 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f10180b3a00 0x7f1018159110 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 >> 172.21.15.150:0/788515821 conn(0x7f101809b4a0 msgr2=0x7f10180b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 shutdown_connections 2024-04-25T14:02:41.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.092+0000 7f101e069640 1 -- 172.21.15.150:0/788515821 wait complete. 2024-04-25T14:02:41.102 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:41.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.213+0000 7fa1c8f7a640 1 Processor -- start 2024-04-25T14:02:41.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.213+0000 7fa1c8f7a640 1 -- start start 2024-04-25T14:02:41.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c8f7a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1c40cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c8f7a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c40b7f40 con 0x7fa1c40b53a0 2024-04-25T14:02:41.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c8f7a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1c4164620 con 0x7fa1c40b8700 2024-04-25T14:02:41.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c8f7a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c4164bc0 con 0x7fa1c4163690 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c1d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa1c40b53a0 0x7fa1c40b57b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41258/0 (socket says 172.21.15.150:41258) 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c1d9b640 1 -- 172.21.15.150:0/2075879675 learned_addr learned my addr 172.21.15.150:0/2075879675 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1299570055 0 0) 0x7fa1c40b7f40 con 0x7fa1c40b53a0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.214+0000 7fa1c259c640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1c40cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1ac003520 con 0x7fa1c40b53a0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 769724646 0 0) 0x7fa1ac003520 con 0x7fa1c40b53a0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 msgr2=0x7fa1c40cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1c40cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 >> v1:172.21.15.187:6789/0 conn(0x7fa1c40b8700 legacy=0x7fa1c40b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c40b3080 con 0x7fa1c40b53a0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fa1b8001670 con 0x7fa1c40b53a0 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 >> v1:172.21.15.150:6789/0 conn(0x7fa1c40b53a0 legacy=0x7fa1c40b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac0039e0 0x7fa1ac003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1ac004d40 0x7fa1ac005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c4164bc0 con 0x7fa1ac004d40 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa1ac00bb80 con 0x7fa1ac0039e0 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1ac00bd80 con 0x7fa1c4163690 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1ac004d40 0x7fa1ac005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c1d9b640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c259c640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac0039e0 0x7fa1ac003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 -- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 msgr2=0x7fa1ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4163690 0x7fa1ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 -- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac0039e0 msgr2=0x7fa1ac003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac0039e0 0x7fa1ac003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.215+0000 7fa1c2d9d640 1 -- 172.21.15.150:0/2075879675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1ac003520 con 0x7fa1ac004d40 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.216+0000 7fa1c2d9d640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1ac004d40 0x7fa1ac005150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40b5ef0 tx=0x7fa1ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6090d3c20663ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.216+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc001370 con 0x7fa1ac004d40 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.216+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1bc0052d0 con 0x7fa1ac004d40 2024-04-25T14:02:41.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.216+0000 7fa1c159a640 1 -- 172.21.15.150:0/2075879675 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc02ab90 con 0x7fa1ac004d40 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2075879675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1ac004d40 msgr2=0x7fa1ac005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2075879675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1ac004d40 0x7fa1ac005150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa1c40b5ef0 tx=0x7fa1ac005830 comp rx=0 tx=0).stop 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c2d9d640 1 -- 172.21.15.150:0/2075879675 reap_dead start 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2075879675 shutdown_connections 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2075879675 >> 172.21.15.150:0/2075879675 conn(0x7fa1c40d03a0 msgr2=0x7fa1c40d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2075879675 shutdown_connections 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.217+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2075879675 wait complete. 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.219+0000 7fa1c8f7a640 1 Processor -- start 2024-04-25T14:02:41.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.219+0000 7fa1c8f7a640 1 -- start start 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac004d40 0x7fa1c4160a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4159ec0 0x7fa1c41e9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c40b2ca0 con 0x7fa1c41586e0 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c40b81e0 con 0x7fa1ac004d40 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c8f7a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa1c40b2e20 con 0x7fa1c4159ec0 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c1d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56466/0 (socket says 172.21.15.150:56466) 2024-04-25T14:02:41.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c1d9b640 1 -- 172.21.15.150:0/2480956115 learned_addr learned my addr 172.21.15.150:0/2480956115 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c259c640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac004d40 0x7fa1c4160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.220+0000 7fa1c2d9d640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4159ec0 0x7fa1c41e9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 -- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4159ec0 msgr2=0x7fa1c41e9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4159ec0 0x7fa1c41e9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 -- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac004d40 msgr2=0x7fa1c4160a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac004d40 0x7fa1c4160a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 -- 172.21.15.150:0/2480956115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c4161490 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c1d9b640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80072f0 tx=0x7fa1b80077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800f960 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1b8018070 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c4161770 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800fd80 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.221+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c4161d30 con 0x7fa1c41586e0 2024-04-25T14:02:41.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.222+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c4076a40 con 0x7fa1c41586e0 2024-04-25T14:02:41.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.223+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa1b801d460 con 0x7fa1c41586e0 2024-04-25T14:02:41.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.223+0000 7fa1b2ffd640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 0x7fa1a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.223+0000 7fa1c259c640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 0x7fa1a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.223+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa1b8095460 con 0x7fa1c41586e0 2024-04-25T14:02:41.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.223+0000 7fa1c259c640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 0x7fa1a40769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4014510 tx=0x7fa1b400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.226 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.228+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1b805f260 con 0x7fa1c41586e0 2024-04-25T14:02:41.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.441+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1c4161220 con 0x7fa1c41586e0 2024-04-25T14:02:41.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.441+0000 7fa1b2ffd640 1 -- 172.21.15.150:0/2480956115 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7fa1b8063560 con 0x7fa1c41586e0 2024-04-25T14:02:41.441 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:41.441 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:41.441 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 msgr2=0x7fa1a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 0x7fa1a40769f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4014510 tx=0x7fa1b400b040 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 msgr2=0x7fa1c4159980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b80072f0 tx=0x7fa1b80077b0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 shutdown_connections 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa1a40744e0 0x7fa1a40769f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa1c4159ec0 0x7fa1c41e9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa1c41586e0 0x7fa1c4159980 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 --2- 172.21.15.150:0/2480956115 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa1ac004d40 0x7fa1c4160a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.443+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 >> 172.21.15.150:0/2480956115 conn(0x7fa1c40d03a0 msgr2=0x7fa1c4163b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.444+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 shutdown_connections 2024-04-25T14:02:41.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.444+0000 7fa1c8f7a640 1 -- 172.21.15.150:0/2480956115 wait complete. 2024-04-25T14:02:41.454 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:41.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.562+0000 7f756f97a640 1 Processor -- start 2024-04-25T14:02:41.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.562+0000 7f756f97a640 1 -- start start 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756f97a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f75680b5ab0 0x7f75680d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756f97a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7568164910 con 0x7f75680b2ca0 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756f97a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75680cfdb0 con 0x7f7568160240 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756f97a640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f75680b80b0 con 0x7f75680b5ab0 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756df17640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f75680b2ca0 0x7f75680b8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41262/0 (socket says 172.21.15.150:41262) 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756df17640 1 -- 172.21.15.150:0/1263572161 learned_addr learned my addr 172.21.15.150:0/1263572161 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:41.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f756cf15640 1 --2- 172.21.15.150:0/1263572161 >> v2:172.21.15.187:3300/0 conn(0x7f75680b5ab0 0x7f75680d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132422302 0 0) 0x7f7568164910 con 0x7f75680b2ca0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7554003520 con 0x7f75680b2ca0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2282159686 0 0) 0x7f7554003520 con 0x7f75680b2ca0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 >> v2:172.21.15.187:3300/0 conn(0x7f75680b5ab0 msgr2=0x7f75680d2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 --2- 172.21.15.150:0/1263572161 >> v2:172.21.15.187:3300/0 conn(0x7f75680b5ab0 0x7f75680d2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 >> v1:172.21.15.187:6789/0 conn(0x7f7568160240 legacy=0x7f75680b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75680d0090 con 0x7f75680b2ca0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f7560001690 con 0x7f75680b2ca0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 >> v1:172.21.15.150:6789/0 conn(0x7f75680b2ca0 legacy=0x7f75680b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75540039b0 0x7f7554003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.564+0000 7f755ffff640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7554004800 0x7f7554004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f755ffff640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f755400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75680b80b0 con 0x7f75540051c0 2024-04-25T14:02:41.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f755400c160 con 0x7f75540039b0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f755400c360 con 0x7f7554004800 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f755400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756d716640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75540039b0 0x7f7554003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756cf15640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7554004800 0x7f7554004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 -- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7554004800 msgr2=0x7f7554004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7554004800 0x7f7554004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 -- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75540039b0 msgr2=0x7f7554003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f75540039b0 0x7f7554003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.565+0000 7f756df17640 1 -- 172.21.15.150:0/1263572161 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7554003520 con 0x7f75540051c0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756df17640 1 -- 172.21.15.150:0/1263572161 reap_dead start 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756df17640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f755400b9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7568164ae0 tx=0x7f7554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2979f9616809754 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560007d10 con 0x7f75540051c0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7560005ad0 con 0x7f75540051c0 2024-04-25T14:02:41.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f755ffff640 1 -- 172.21.15.150:0/1263572161 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7560007070 con 0x7f75540051c0 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756f97a640 1 -- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 msgr2=0x7f755400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756f97a640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f755400b9c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7568164ae0 tx=0x7f7554005830 comp rx=0 tx=0).stop 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756f97a640 1 -- 172.21.15.150:0/1263572161 shutdown_connections 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756f97a640 1 --2- 172.21.15.150:0/1263572161 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f755400b9c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.566+0000 7f756f97a640 1 -- 172.21.15.150:0/1263572161 >> 172.21.15.150:0/1263572161 conn(0x7f75680d03a0 msgr2=0x7f75680d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.567+0000 7f756f97a640 1 -- 172.21.15.150:0/1263572161 shutdown_connections 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.567+0000 7f756f97a640 1 -- 172.21.15.150:0/1263572161 wait complete. 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.568+0000 7f756f97a640 1 Processor -- start 2024-04-25T14:02:41.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.568+0000 7f756f97a640 1 -- start start 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756814e940 0x7f756814ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75680cb1b0 0x7f75680cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f75680d2ee0 con 0x7f75540051c0 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f75680d2d60 con 0x7f756814e940 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f75680d3060 con 0x7f75680cb1b0 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56492/0 (socket says 172.21.15.150:56492) 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 -- 172.21.15.150:0/2208969324 learned_addr learned my addr 172.21.15.150:0/2208969324 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756df17640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75680cb1b0 0x7f75680cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 -- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75680cb1b0 msgr2=0x7f75680cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75680cb1b0 0x7f75680cb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756cf15640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756814e940 0x7f756814ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 -- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756814e940 msgr2=0x7f756814ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756814e940 0x7f756814ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 -- 172.21.15.150:0/2208969324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f756814f290 con 0x7f75540051c0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756d716640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f756400f960 tx=0x7f756400fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7564003b80 con 0x7f75540051c0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7564018070 con 0x7f75540051c0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f756806e690 con 0x7f75540051c0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.570+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f756400d790 con 0x7f75540051c0 2024-04-25T14:02:41.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.571+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f756814cd10 con 0x7f75540051c0 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.571+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7568076a40 con 0x7f75540051c0 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.572+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f756400d930 con 0x7f75540051c0 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.572+0000 7f755dffb640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 0x7f754c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.573+0000 7f756cf15640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 0x7f754c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.573+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f7564095640 con 0x7f75540051c0 2024-04-25T14:02:41.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.573+0000 7f756cf15640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 0x7f754c0769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75680b3c70 tx=0x7f7558008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.577+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f756814f290 con 0x7f75540051c0 2024-04-25T14:02:41.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.756+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f756815fe00 con 0x7f75540051c0 2024-04-25T14:02:41.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.758+0000 7f755dffb640 1 -- 172.21.15.150:0/2208969324 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) ==== 74+0+18514 (secure 0 0 0) 0x7f756405fb20 con 0x7f75540051c0 2024-04-25T14:02:41.757 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:41.757 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:13.547681+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:41.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 msgr2=0x7f754c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 0x7f754c0769c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f75680b3c70 tx=0x7f7558008040 comp rx=0 tx=0).stop 2024-04-25T14:02:41.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 msgr2=0x7f75680b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f756400f960 tx=0x7f756400fe20 comp rx=0 tx=0).stop 2024-04-25T14:02:41.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 shutdown_connections 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f754c0744b0 0x7f754c0769c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f75680cb1b0 0x7f75680cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f756814e940 0x7f756814ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 --2- 172.21.15.150:0/2208969324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f75540051c0 0x7f75680b2490 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 >> 172.21.15.150:0/2208969324 conn(0x7f75680d03a0 msgr2=0x7f75680b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 shutdown_connections 2024-04-25T14:02:41.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.760+0000 7f756f97a640 1 -- 172.21.15.150:0/2208969324 wait complete. 2024-04-25T14:02:41.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:02:41.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.879+0000 7f686b8f0640 1 Processor -- start 2024-04-25T14:02:41.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.879+0000 7f686b8f0640 1 -- start start 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.880+0000 7f686b8f0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68640d2e90 0x7f6864160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686b8f0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68640b53a0 0x7f68640b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686b8f0640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68640b5cf0 con 0x7f68640b2b10 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686b8f0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f68640b7db0 con 0x7f68640d2e90 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686b8f0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6864164690 con 0x7f68640b53a0 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68640d2e90 0x7f6864160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686968c640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f68640b2b10 0x7f68640cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:59168/0 (socket says 172.21.15.150:59168) 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f68640d2e90 0x7f6864160220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:38728/0 (socket says 172.21.15.150:38728) 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6869e8d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68640b53a0 0x7f68640b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f686968c640 1 -- 172.21.15.150:0/4260795524 learned_addr learned my addr 172.21.15.150:0/4260795524 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 -- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68640b53a0 msgr2=0x7f68640b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 --2- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68640b53a0 0x7f68640b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 -- 172.21.15.150:0/4260795524 >> v1:172.21.15.187:6789/0 conn(0x7f68640b2b10 legacy=0x7f68640cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.881+0000 7f6868e8b640 1 -- 172.21.15.150:0/4260795524 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6864164bc0 con 0x7f68640d2e90 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.882+0000 7f6868e8b640 1 --2- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f6864160220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68580097a0 tx=0x7f685802a060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54f740d715a820e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.882+0000 7f6863fff640 1 -- 172.21.15.150:0/4260795524 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858037070 con 0x7f68640d2e90 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.882+0000 7f6863fff640 1 -- 172.21.15.150:0/4260795524 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685802adb0 con 0x7f68640d2e90 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.882+0000 7f6863fff640 1 -- 172.21.15.150:0/4260795524 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6858032660 con 0x7f68640d2e90 2024-04-25T14:02:41.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 -- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 msgr2=0x7f6864160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 --2- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f6864160220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f68580097a0 tx=0x7f685802a060 comp rx=0 tx=0).stop 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 -- 172.21.15.150:0/4260795524 shutdown_connections 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 --2- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68640b53a0 0x7f68640b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 --2- 172.21.15.150:0/4260795524 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f6864160220 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 -- 172.21.15.150:0/4260795524 >> 172.21.15.150:0/4260795524 conn(0x7f68640d03a0 msgr2=0x7f68640d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 -- 172.21.15.150:0/4260795524 shutdown_connections 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.883+0000 7f686b8f0640 1 -- 172.21.15.150:0/4260795524 wait complete. 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.885+0000 7f686b8f0640 1 Processor -- start 2024-04-25T14:02:41.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.885+0000 7f686b8f0640 1 -- start start 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f686406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6864162060 0x7f6864162470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68641648e0 con 0x7f68640b53a0 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f68640b8350 con 0x7f68640d2e90 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686b8f0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f68640b7fe0 con 0x7f6864162060 2024-04-25T14:02:41.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56504/0 (socket says 172.21.15.150:56504) 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 -- 172.21.15.150:0/3602381347 learned_addr learned my addr 172.21.15.150:0/3602381347 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f6868e8b640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f686406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f6869e8d640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6864162060 0x7f6864162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 -- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6864162060 msgr2=0x7f6864162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6864162060 0x7f6864162470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 -- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 msgr2=0x7f686406e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f686406e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.886+0000 7f686968c640 1 -- 172.21.15.150:0/3602381347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68641629b0 con 0x7f68640b53a0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f686968c640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f685400ca70 tx=0x7f685400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854017070 con 0x7f68640b53a0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f685400abd0 con 0x7f68640b53a0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f686406ee60 con 0x7f68640b53a0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6854012400 con 0x7f68640b53a0 2024-04-25T14:02:41.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.887+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f686406f650 con 0x7f68640b53a0 2024-04-25T14:02:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.888+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6864076a40 con 0x7f68640b53a0 2024-04-25T14:02:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.888+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f685401a070 con 0x7f68640b53a0 2024-04-25T14:02:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.889+0000 7f6861ffb640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 0x7f68500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:41.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.889+0000 7f6868e8b640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 0x7f68500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:41.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.889+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f6854095220 con 0x7f68640b53a0 2024-04-25T14:02:41.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.889+0000 7f6868e8b640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 0x7f68500769c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6858034000 tx=0x7f685802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:41.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:41.893+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68641629b0 con 0x7f68640b53a0 2024-04-25T14:02:42.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.103+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68641643d0 con 0x7f68640b53a0 2024-04-25T14:02:42.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.104+0000 7f6861ffb640 1 -- 172.21.15.150:0/3602381347 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f68640b5cf0 con 0x7f68640b53a0 2024-04-25T14:02:42.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:42.103 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:02:42.103 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:02:42.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 msgr2=0x7f68500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 0x7f68500769c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6858034000 tx=0x7f685802e000 comp rx=0 tx=0).stop 2024-04-25T14:02:42.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 msgr2=0x7f686414bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f685400ca70 tx=0x7f685400cf30 comp rx=0 tx=0).stop 2024-04-25T14:02:42.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 shutdown_connections 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68500744b0 0x7f68500769c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6864162060 0x7f6864162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68640d2e90 0x7f686406e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 --2- 172.21.15.150:0/3602381347 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68640b53a0 0x7f686414bfc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 >> 172.21.15.150:0/3602381347 conn(0x7f68640d03a0 msgr2=0x7f68640ce8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 shutdown_connections 2024-04-25T14:02:42.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.106+0000 7f686b8f0640 1 -- 172.21.15.150:0/3602381347 wait complete. 2024-04-25T14:02:42.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:02:08.490370+0000', 'modified': '2024-04-25T14:02:30.598800+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/3860691734', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 3860691734}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 3860691734}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-04-25T14:02:42.519 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:02:42.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T14:02:42.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.639+0000 7fe01aa14640 1 Processor -- start 2024-04-25T14:02:42.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.639+0000 7fe01aa14640 1 -- start start 2024-04-25T14:02:42.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe01aa14640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0141610f0 0x7fe014167d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe01aa14640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0140cf300 con 0x7fe0140d03a0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe01aa14640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0140cf5e0 con 0x7fe014070820 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe01aa14640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe0140cff10 con 0x7fe0141610f0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe013fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe0140d03a0 0x7fe0140cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41270/0 (socket says 172.21.15.150:41270) 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe013fff640 1 -- 172.21.15.150:0/926525712 learned_addr learned my addr 172.21.15.150:0/926525712 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe018fb1640 1 --2- 172.21.15.150:0/926525712 >> v2:172.21.15.187:3300/0 conn(0x7fe0141610f0 0x7fe014167d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3282523098 0 0) 0x7fe0140cf300 con 0x7fe0140d03a0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.640+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdffc003520 con 0x7fe0140d03a0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158451737 0 0) 0x7fe0140cf5e0 con 0x7fe014070820 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0140cf300 con 0x7fe014070820 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1791623649 0 0) 0x7fdffc003520 con 0x7fe0140d03a0 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 >> v2:172.21.15.187:3300/0 conn(0x7fe0141610f0 msgr2=0x7fe014167d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 --2- 172.21.15.150:0/926525712 >> v2:172.21.15.187:3300/0 conn(0x7fe0141610f0 0x7fe014167d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 >> v1:172.21.15.187:6789/0 conn(0x7fe014070820 legacy=0x7fe0141609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0140cf8c0 con 0x7fe0140d03a0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fe004001670 con 0x7fe0140d03a0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 >> v1:172.21.15.150:6789/0 conn(0x7fe0140d03a0 legacy=0x7fe0140cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdffc003b40 0x7fdffc003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdffc004800 0x7fdffc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fdffc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0140cff10 con 0x7fdffc0051a0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc00c140 con 0x7fdffc003b40 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc00c340 con 0x7fdffc004800 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fdffc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe013fff640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdffc003b40 0x7fdffc003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe0137fe640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdffc004800 0x7fdffc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 -- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdffc004800 msgr2=0x7fdffc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdffc004800 0x7fdffc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 -- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdffc003b40 msgr2=0x7fdffc003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdffc003b40 0x7fdffc003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.641+0000 7fe018fb1640 1 -- 172.21.15.150:0/926525712 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc003520 con 0x7fdffc0051a0 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.642+0000 7fe018fb1640 1 -- 172.21.15.150:0/926525712 reap_dead start 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.642+0000 7fe018fb1640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fdffc00b9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0057e0 tx=0x7fdffc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b687c3a03c5c7651 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.642+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c03a070 con 0x7fdffc0051a0 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.642+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00c02dab0 con 0x7fdffc0051a0 2024-04-25T14:02:42.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.642+0000 7fe012ffd640 1 -- 172.21.15.150:0/926525712 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c009d00 con 0x7fdffc0051a0 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 -- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 msgr2=0x7fdffc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fdffc00b9c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fdffc0057e0 tx=0x7fdffc005830 comp rx=0 tx=0).stop 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 -- 172.21.15.150:0/926525712 shutdown_connections 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 --2- 172.21.15.150:0/926525712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fdffc00b9c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 -- 172.21.15.150:0/926525712 >> 172.21.15.150:0/926525712 conn(0x7fe0140ce390 msgr2=0x7fe0140ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 -- 172.21.15.150:0/926525712 shutdown_connections 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.643+0000 7fe01aa14640 1 -- 172.21.15.150:0/926525712 wait complete. 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.645+0000 7fe01aa14640 1 Processor -- start 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.645+0000 7fe01aa14640 1 -- start start 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0140b6760 0x7fe0141f9950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0141f9e90 0x7fe0140af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0140cde20 con 0x7fdffc0051a0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0140cdca0 con 0x7fe0141f9e90 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.646+0000 7fe01aa14640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0140cdfa0 con 0x7fe0140b6760 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56512/0 (socket says 172.21.15.150:56512) 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 -- 172.21.15.150:0/2823669086 learned_addr learned my addr 172.21.15.150:0/2823669086 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0140b6760 0x7fe0141f9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe018fb1640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0141f9e90 0x7fe0140af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 -- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0140b6760 msgr2=0x7fe0141f9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0140b6760 0x7fe0141f9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 -- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0141f9e90 msgr2=0x7fe0140af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0141f9e90 0x7fe0140af5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 -- 172.21.15.150:0/2823669086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0140b0650 con 0x7fdffc0051a0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe013fff640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe004007340 tx=0x7fe004007740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe010ff9640 1 -- 172.21.15.150:0/2823669086 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe004001610 con 0x7fdffc0051a0 2024-04-25T14:02:42.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe010ff9640 1 -- 172.21.15.150:0/2823669086 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe004018070 con 0x7fdffc0051a0 2024-04-25T14:02:42.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0140b62b0 con 0x7fdffc0051a0 2024-04-25T14:02:42.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe010ff9640 1 -- 172.21.15.150:0/2823669086 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00400fb00 con 0x7fdffc0051a0 2024-04-25T14:02:42.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.647+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0140af0b0 con 0x7fdffc0051a0 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.650+0000 7fe010ff9640 1 -- 172.21.15.150:0/2823669086 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe00400fca0 con 0x7fdffc0051a0 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.650+0000 7fe010ff9640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 0x7fe0000769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.650+0000 7fe010ff9640 1 -- 172.21.15.150:0/2823669086 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe0040953a0 con 0x7fdffc0051a0 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.650+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 0x7fe0000769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.651+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 0x7fe0000769f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe0140b6c60 tx=0x7fe008009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.651+0000 7fe02199b740 1 Processor -- start 2024-04-25T14:02:42.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.651+0000 7fe02199b740 1 -- start start 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d233600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce46590 0x557e7d209260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557e7d16d740 con 0x557e7ce3e3b0 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557e7d1093a0 con 0x557e7ce67270 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557e7d169700 con 0x557e7ce46590 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56518/0 (socket says 172.21.15.150:56518) 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 -- 172.21.15.150:0/2248267780 learned_addr learned my addr 172.21.15.150:0/2248267780 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe013fff640 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d233600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe018fb1640 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce46590 0x557e7d209260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 -- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce46590 msgr2=0x557e7d209260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce46590 0x557e7d209260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 -- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 msgr2=0x557e7d233600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d233600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.652+0000 7fe0137fe640 1 -- 172.21.15.150:0/2248267780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e7d125750 con 0x557e7ce3e3b0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.653+0000 7fe0137fe640 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe008012c10 tx=0x7fe008015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d579632252d4baef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.653+0000 7fdff9ffb640 1 -- 172.21.15.150:0/2248267780 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008019070 con 0x557e7ce3e3b0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.653+0000 7fdff9ffb640 1 -- 172.21.15.150:0/2248267780 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe008026070 con 0x557e7ce3e3b0 2024-04-25T14:02:42.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.653+0000 7fdff9ffb640 1 -- 172.21.15.150:0/2248267780 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008015bc0 con 0x557e7ce3e3b0 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 -- 172.21.15.150:0/2248267780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 msgr2=0x557e7d22ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe008012c10 tx=0x7fe008015000 comp rx=0 tx=0).stop 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 -- 172.21.15.150:0/2248267780 shutdown_connections 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce46590 0x557e7d209260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce3e3b0 0x557e7d22ffa0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 --2- 172.21.15.150:0/2248267780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d233600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 -- 172.21.15.150:0/2248267780 >> 172.21.15.150:0/2248267780 conn(0x557e7ce5b3c0 msgr2=0x557e7d2089f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 -- 172.21.15.150:0/2248267780 shutdown_connections 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.654+0000 7fe02199b740 1 -- 172.21.15.150:0/2248267780 wait complete. 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.655+0000 7fe02199b740 1 -- start start 2024-04-25T14:02:42.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.655+0000 7fe02199b740 1 Processor -- start 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d230260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d2307a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557e7d0f4fe0 con 0x557e7ce67270 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557e7d235ea0 con 0x557e7ce46590 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557e7d24bd40 con 0x557e7ce3e3b0 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe02199b740 10 client.0 _ll_register_callbacks cb 0x557e7d10b150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe018fb1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe018fb1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56532/0 (socket says 172.21.15.150:56532) 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe018fb1640 1 -- 172.21.15.150:0/3670344992 learned_addr learned my addr 172.21.15.150:0/3670344992 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe0137fe640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d2307a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.656+0000 7fe013fff640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d230260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 -- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 msgr2=0x557e7d230260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d230260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 -- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 msgr2=0x557e7d2307a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d2307a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 -- 172.21.15.150:0/3670344992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e7ce7f2d0 con 0x557e7ce67270 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe00c010510 tx=0x7fe00c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a696e81ee0696c62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c045060 con 0x557e7ce67270 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe00c03f680 con 0x557e7ce67270 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe008026070 con 0x557e7ce67270 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe00c04b050 con 0x557e7ce67270 2024-04-25T14:02:42.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe02199b740 10 client.4848 fetch_fsmap learned FSMap version 8 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x557e7cf496b0 con 0x557e7ce67270 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe008026070 con 0x557e7ce67270 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe00c04e020 con 0x557e7ce67270 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.657+0000 7fe02199b740 10 client.4848 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe02199b740 10 client.4848 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe02199b740 20 client.4848 populate_metadata read hostname 'smithi150' 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7ce25e40 0x557e7d232180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe02199b740 4 client.4848 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 --> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e7d17b350 con 0x557e7ce25e40 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe013fff640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7ce25e40 0x557e7d232180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fdff2ffd640 10 client.4848 ms_handle_connect on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.658+0000 7fe013fff640 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7ce25e40 0x557e7d232180 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a72aa068a26a905e server_cookie=fd016fc0de9eab7 in_seq=0 out_seq=0 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fdff2ffd640 1 -- 172.21.15.150:0/3670344992 <== mds.0 v2:172.21.15.187:6833/3860691734 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557e7d17b350 con 0x557e7ce25e40 2024-04-25T14:02:42.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fdff2ffd640 10 client.4848 handle_command_reply: tid=0 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 client.4848 shutdown 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 msgr2=0x557e7d260960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe00c010510 tx=0x7fe00c004ab0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 shutdown_connections 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d260960 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7ce25e40 0x557e7d232180 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d2307a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 --2- 172.21.15.150:0/3670344992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d230260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fdff2ffd640 0 client.4848 ms_handle_reset on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 >> 172.21.15.150:0/3670344992 conn(0x557e7ce5b3c0 msgr2=0x557e7d2089f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 shutdown_connections 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 1 -- 172.21.15.150:0/3670344992 wait complete. 2024-04-25T14:02:42.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.659+0000 7fe02199b740 20 client.4848 trim_cache size 0 max 16384 2024-04-25T14:02:42.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.678+0000 7fe02199b740 1 Processor -- start 2024-04-25T14:02:42.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.678+0000 7fe02199b740 1 -- start start 2024-04-25T14:02:42.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d232370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d207320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe014203380 con 0x557e7ce67270 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0142026e0 con 0x557e7ce46590 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe014202d30 con 0x557e7ce3e3b0 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.679+0000 7fe018fb1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56544/0 (socket says 172.21.15.150:56544) 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 -- 172.21.15.150:0/3631466646 learned_addr learned my addr 172.21.15.150:0/3631466646 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe013fff640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d232370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe0137fe640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d207320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 -- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 msgr2=0x557e7d232370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d232370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 -- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 msgr2=0x557e7d207320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d207320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 -- 172.21.15.150:0/3631466646 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e7d232d20 con 0x557e7ce67270 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe013fff640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d232370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fe018fb1640 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b080 tx=0x7fe00c03f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dbaa85bbc4f374 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fdff97fa640 1 -- 172.21.15.150:0/3631466646 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c03a040 con 0x557e7ce67270 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fdff97fa640 1 -- 172.21.15.150:0/3631466646 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00c03f480 con 0x557e7ce67270 2024-04-25T14:02:42.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.680+0000 7fdff97fa640 1 -- 172.21.15.150:0/3631466646 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe00c03f870 con 0x557e7ce67270 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 -- 172.21.15.150:0/3631466646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 msgr2=0x557e7d207860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe00c00b080 tx=0x7fe00c03f000 comp rx=0 tx=0).stop 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 -- 172.21.15.150:0/3631466646 shutdown_connections 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce67270 0x557e7d207860 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce46590 0x557e7d207320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 --2- 172.21.15.150:0/3631466646 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d232370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 -- 172.21.15.150:0/3631466646 >> 172.21.15.150:0/3631466646 conn(0x557e7ce5b3c0 msgr2=0x557e7d24b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 -- 172.21.15.150:0/3631466646 shutdown_connections 2024-04-25T14:02:42.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.681+0000 7fe02199b740 1 -- 172.21.15.150:0/3631466646 wait complete. 2024-04-25T14:02:42.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.682+0000 7fe02199b740 1 -- start start 2024-04-25T14:02:42.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.682+0000 7fe02199b740 1 Processor -- start 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d236790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d207720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x557e7d2562a0 con 0x557e7ce46590 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x557e7d2357a0 con 0x557e7ce67270 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.683+0000 7fe02199b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x557e7d235060 con 0x557e7ce3e3b0 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe02199b740 10 client.0 _ll_register_callbacks cb 0x7fdffc0036f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56558/0 (socket says 172.21.15.150:56558) 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 -- 172.21.15.150:0/1722516856 learned_addr learned my addr 172.21.15.150:0/1722516856 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:42.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe013fff640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d236790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe018fb1640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d207720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 -- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 msgr2=0x557e7d236790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d236790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 -- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 msgr2=0x557e7d207720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d207720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 -- 172.21.15.150:0/1722516856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e7d24b440 con 0x557e7ce46590 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe013fff640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d236790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe0137fe640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe008001740 tx=0x7fe0080012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4707ab97f507b165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fdff2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe008007600 con 0x557e7ce46590 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fe008005020 con 0x557e7ce46590 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.684+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe00c03f480 con 0x557e7ce46590 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7fe00803f050 con 0x557e7ce46590 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 10 client.4854 fetch_fsmap learned FSMap version 8 2024-04-25T14:02:42.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x557e7d2cb3d0 con 0x557e7ce46590 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe00c03f480 con 0x557e7ce46590 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7fe008005220 con 0x557e7ce46590 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 10 client.4854 fetch_fsmap finished waiting for FSMap version 8 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 10 client.4854 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 20 client.4854 populate_metadata read hostname 'smithi150' 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7d202300 0x557e7d202710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 4 client.4854 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 --> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] -- command(tid 0: {"prefix": "status"}) -- 0x557e7d1751c0 con 0x557e7d202300 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fe013fff640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7d202300 0x557e7d202710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.685+0000 7fdff2ffd640 10 client.4854 ms_handle_connect on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe013fff640 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7d202300 0x557e7d202710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32ca3af7aaa26970 server_cookie=2147642026dff6af in_seq=0 out_seq=0 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fdff2ffd640 1 -- 172.21.15.150:0/1722516856 <== mds.0 v2:172.21.15.187:6833/3860691734 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x557e7d1751c0 con 0x557e7d202300 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fdff2ffd640 10 client.4854 handle_command_reply: tid=0 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 client.4854 shutdown 2024-04-25T14:02:42.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 msgr2=0x557e7d2ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe008001740 tx=0x7fe0080012d0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 shutdown_connections 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:6833/3860691734,v1:172.21.15.187:6835/3860691734] conn(0x557e7d202300 0x557e7d202710 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x557e7ce67270 0x557e7d207720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x557e7ce46590 0x557e7d2ca970 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 --2- 172.21.15.150:0/1722516856 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x557e7ce3e3b0 0x557e7d236790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 >> 172.21.15.150:0/1722516856 conn(0x557e7ce5b3c0 msgr2=0x557e7d232470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fdff2ffd640 0 client.4854 ms_handle_reset on v2:172.21.15.187:6833/3860691734 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 shutdown_connections 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 1 -- 172.21.15.150:0/1722516856 wait complete. 2024-04-25T14:02:42.690 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.686+0000 7fe02199b740 20 client.4854 trim_cache size 0 max 16384 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 4293, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 33.532223868000003, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 8, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 24, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 24, 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 40.384878514999997 2024-04-25T14:02:42.691 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:02:42.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 msgr2=0x7fe0000769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 0x7fe0000769f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe0140b6c60 tx=0x7fe008009040 comp rx=0 tx=0).stop 2024-04-25T14:02:42.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 msgr2=0x7fe014160830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe004007340 tx=0x7fe004007740 comp rx=0 tx=0).stop 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 shutdown_connections 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0000744e0 0x7fe0000769f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0141f9e90 0x7fe0140af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0140b6760 0x7fe0141f9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 --2- 172.21.15.150:0/2823669086 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdffc0051a0 0x7fe014160830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 >> 172.21.15.150:0/2823669086 conn(0x7fe0140ce390 msgr2=0x7fe01406f750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 shutdown_connections 2024-04-25T14:02:42.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:42.687+0000 7fe01aa14640 1 -- 172.21.15.150:0/2823669086 wait complete. 2024-04-25T14:02:42.697 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T14:02:42.706 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T14:02:42.706 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T14:02:42.707 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T14:02:42.707 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T14:02:42.756 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T14:02:42.756 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-25T14:02:42.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:42.757 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:02:42.772 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-25T14:02:42.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:42.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-25T14:02:42.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:42.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-04-25T14:02:42.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.324 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:02:43.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.343 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-04-25T14:02:43.409 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-04-25T14:02:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:43.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.411 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:02:43.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.430 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-25T14:02:43.501 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-25T14:02:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:43.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.503 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:02:43.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.559 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-25T14:02:43.625 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-25T14:02:43.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:02:43.627 INFO:tasks.kclient:client.0 config is {'syntax': 'v2'} 2024-04-25T14:02:43.627 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T14:02:43.627 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T14:02:43.627 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:02:43.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.737+0000 7fc03cc92640 1 Processor -- start 2024-04-25T14:02:43.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.737+0000 7fc03cc92640 1 -- start start 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03cc92640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 0x7fc038070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03cc92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc038154d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03cc92640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0380aec30 con 0x7fc038161b10 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03cc92640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc0380d1530 con 0x7fc0380d1120 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03cc92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0380aef10 con 0x7fc038154950 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc035d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc038161b10 0x7fc038161f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41282/0 (socket says 172.21.15.150:41282) 2024-04-25T14:02:43.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc035d9b640 1 -- 172.21.15.150:0/2215243945 learned_addr learned my addr 172.21.15.150:0/2215243945 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc036d9d640 1 --2- 172.21.15.150:0/2215243945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc038154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.738+0000 7fc03659c640 1 --2- 172.21.15.150:0/2215243945 >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 0x7fc038070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc03559a640 1 -- 172.21.15.150:0/2215243945 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3133935720 0 0) 0x7fc0380aec30 con 0x7fc038161b10 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc03559a640 1 -- 172.21.15.150:0/2215243945 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc020003520 con 0x7fc038161b10 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc036d9d640 1 -- 172.21.15.150:0/2215243945 >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 msgr2=0x7fc038070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc036d9d640 1 --2- 172.21.15.150:0/2215243945 >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 0x7fc038070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc036d9d640 1 -- 172.21.15.150:0/2215243945 >> v1:172.21.15.150:6789/0 conn(0x7fc038161b10 legacy=0x7fc038161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc036d9d640 1 -- 172.21.15.150:0/2215243945 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0380af1f0 con 0x7fc038154950 2024-04-25T14:02:43.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc03659c640 1 --2- 172.21.15.150:0/2215243945 >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 0x7fc038070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:43.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.739+0000 7fc036d9d640 1 --2- 172.21.15.150:0/2215243945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc038154d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0300097a0 tx=0x7fc03002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bab4b4577087b47f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:43.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03559a640 1 -- 172.21.15.150:0/2215243945 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030037070 con 0x7fc038154950 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03559a640 1 -- 172.21.15.150:0/2215243945 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc03002adb0 con 0x7fc038154950 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03559a640 1 -- 172.21.15.150:0/2215243945 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030032660 con 0x7fc038154950 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 -- 172.21.15.150:0/2215243945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 msgr2=0x7fc038154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 --2- 172.21.15.150:0/2215243945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc038154d60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc0300097a0 tx=0x7fc03002a060 comp rx=0 tx=0).stop 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 -- 172.21.15.150:0/2215243945 shutdown_connections 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 --2- 172.21.15.150:0/2215243945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc038154d60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 --2- 172.21.15.150:0/2215243945 >> v2:172.21.15.187:3300/0 conn(0x7fc0380d1120 0x7fc038070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.740+0000 7fc03cc92640 1 -- 172.21.15.150:0/2215243945 >> 172.21.15.150:0/2215243945 conn(0x7fc0380d1ae0 msgr2=0x7fc0380d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.741+0000 7fc03cc92640 1 -- 172.21.15.150:0/2215243945 shutdown_connections 2024-04-25T14:02:43.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.741+0000 7fc03cc92640 1 -- 172.21.15.150:0/2215243945 wait complete. 2024-04-25T14:02:43.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.742+0000 7fc03cc92640 1 Processor -- start 2024-04-25T14:02:43.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.743+0000 7fc03cc92640 1 -- start start 2024-04-25T14:02:43.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0380d1120 0x7fc03814c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc0380b4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc0381552a0 con 0x7fc038161b10 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc038071430 con 0x7fc0380d1120 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03cc92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc0381588f0 con 0x7fc038154950 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56570/0 (socket says 172.21.15.150:56570) 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 -- 172.21.15.150:0/3726007829 learned_addr learned my addr 172.21.15.150:0/3726007829 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc035d9b640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc0380b4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc03659c640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0380d1120 0x7fc03814c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 -- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 msgr2=0x7fc0380b4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc0380b4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 -- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0380d1120 msgr2=0x7fc03814c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0380d1120 0x7fc03814c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 -- 172.21.15.150:0/3726007829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0380b5b80 con 0x7fc038161b10 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.744+0000 7fc036d9d640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098c0 tx=0x7fc0300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.745+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030041020 con 0x7fc038161b10 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.745+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc03003c670 con 0x7fc038161b10 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.745+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc030037040 con 0x7fc038161b10 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.745+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc038156d90 con 0x7fc038161b10 2024-04-25T14:02:43.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.745+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc03814cf30 con 0x7fc038161b10 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.746+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03814d160 con 0x7fc038161b10 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.746+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc030044020 con 0x7fc038161b10 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.747+0000 7fc026ffd640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 0x7fc0180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.747+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fc0300b5b80 con 0x7fc038161b10 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.747+0000 7fc03659c640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 0x7fc0180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:43.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.747+0000 7fc03659c640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 0x7fc0180769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc028009bd0 tx=0x7fc02800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:43.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.751+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc03814d160 con 0x7fc038161b10 2024-04-25T14:02:43.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.965+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fc0380b5910 con 0x7fc038161b10 2024-04-25T14:02:43.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.966+0000 7fc026ffd640 1 -- 172.21.15.150:0/3726007829 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fc03004b030 con 0x7fc038161b10 2024-04-25T14:02:43.965 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T14:02:43.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 msgr2=0x7fc0180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 0x7fc0180769c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc028009bd0 tx=0x7fc02800b040 comp rx=0 tx=0).stop 2024-04-25T14:02:43.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 msgr2=0x7fc0380b5340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:43.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc0300098c0 tx=0x7fc0300046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 shutdown_connections 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc0180744b0 0x7fc0180769c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc038161b10 0x7fc0380b5340 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc038154950 0x7fc0380b4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 --2- 172.21.15.150:0/3726007829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc0380d1120 0x7fc03814c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 >> 172.21.15.150:0/3726007829 conn(0x7fc0380d1ae0 msgr2=0x7fc0380b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 shutdown_connections 2024-04-25T14:02:43.971 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:43.968+0000 7fc03cc92640 1 -- 172.21.15.150:0/3726007829 wait complete. 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:02:43.978 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:02:43.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:43.979 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-04-25T14:02:43.993 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:43.993 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:02:43.994 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:02:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42736sec preferred_lft 42736sec 2024-04-25T14:02:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::eb5b:7bab:e07b:2aac/64 scope link noprefixroute 2024-04-25T14:02:43.995 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:43.995 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add name ceph-brx type bridge 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr flush dev ceph-brx 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set ceph-brx up 2024-04-25T14:02:43.995 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T14:02:43.996 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:44.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:44.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:44.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.129 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:02:44.151 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-25T14:02:44.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.152 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-04-25T14:02:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-04-25T14:02:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:02:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:02:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:02:44.209 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:02:44.210 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:44.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:44.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:44.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.333 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:02:44.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.347 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-04-25T14:02:44.402 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:44.402 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:44.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T14:02:44.403 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T14:02:44.403 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:44.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:44.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:44.518 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T14:02:44.519 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:44.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:44.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-04-25T14:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T14:02:44.628 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:44.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:44.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:44.702 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:44.702 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T14:02:44.717 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:02:44.717 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:44.717 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:02:44.771 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T14:02:44.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:44.862 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:44.863 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:44.972 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:44.973 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:44.973 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:02:44.973 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with new device syntax: 0@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:02:44.973 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:02:44.974 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T14:02:44.974 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:02:44.974 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:02:44.974 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:02:44.974 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:02:44.976 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:02:44.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:44.976 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:02:44.999 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:45.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:45.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:45.096 INFO:teuthology.orchestra.run.smithi150.stdout:client4810 2024-04-25T14:02:45.096 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:45.096 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:45.112 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:45.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:45.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:45.209 INFO:teuthology.orchestra.run.smithi150.stdout:client4810 2024-04-25T14:02:45.210 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:45.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:45.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:45.265 INFO:teuthology.orchestra.run.smithi150.stdout:client4810 2024-04-25T14:02:45.266 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/77163e42-55ae-4cab-b506-bd8c531cc087.client4810/status') 2024-04-25T14:02:45.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:45.322 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.4810 (3)172.21.15.150:0/1872398096 2024-04-25T14:02:45.323 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2024-04-25T14:02:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2024-04-25T14:02:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2024-04-25T14:02:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:71 bytes copied, 2.6548e-05 s, 2.7 MB/s 2024-04-25T14:02:45.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:45.330 INFO:tasks.kclient:client.2 config is {'syntax': 'v2'} 2024-04-25T14:02:45.330 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T14:02:45.331 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T14:02:45.331 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:02:45.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.439+0000 7f97d6a8c640 1 Processor -- start 2024-04-25T14:02:45.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.439+0000 7f97d6a8c640 1 -- start start 2024-04-25T14:02:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.440+0000 7f97d6a8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97d006d4d0 0x7f97d00736d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.440+0000 7f97d6a8c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d015f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.440 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97d6a8c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d0074020 con 0x7f97d015f150 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97d6a8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f97d00d1280 con 0x7f97d006d4d0 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97d6a8c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97d006d8e0 con 0x7f97d0073c10 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97d006d4d0 0x7f97d00736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f97d006d4d0 0x7f97d00736d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56576/0 (socket says 172.21.15.150:56576) 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 -- 172.21.15.150:0/2782192912 learned_addr learned my addr 172.21.15.150:0/2782192912 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97d5029640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d015f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 -- 172.21.15.150:0/2782192912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 msgr2=0x7f97d015f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d015f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 -- 172.21.15.150:0/2782192912 >> v1:172.21.15.187:6789/0 conn(0x7f97d015f150 legacy=0x7f97d00cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:45.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.441+0000 7f97cffff640 1 -- 172.21.15.150:0/2782192912 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d006c710 con 0x7f97d006d4d0 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.442+0000 7f97cffff640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d00736d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009870 tx=0x7f97c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7913249a4b456899 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.442+0000 7f97cf7fe640 1 -- 172.21.15.150:0/2782192912 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4037070 con 0x7f97d006d4d0 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.442+0000 7f97cf7fe640 1 -- 172.21.15.150:0/2782192912 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c4032440 con 0x7f97d006d4d0 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.442+0000 7f97cf7fe640 1 -- 172.21.15.150:0/2782192912 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c4032810 con 0x7f97d006d4d0 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 -- 172.21.15.150:0/2782192912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 msgr2=0x7f97d00736d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d00736d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f97c4009870 tx=0x7f97c402a270 comp rx=0 tx=0).stop 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 -- 172.21.15.150:0/2782192912 shutdown_connections 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d015f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/2782192912 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d00736d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 -- 172.21.15.150:0/2782192912 >> 172.21.15.150:0/2782192912 conn(0x7f97d00d1b40 msgr2=0x7f97d00d1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 -- 172.21.15.150:0/2782192912 shutdown_connections 2024-04-25T14:02:45.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.443+0000 7f97d6a8c640 1 -- 172.21.15.150:0/2782192912 wait complete. 2024-04-25T14:02:45.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.445+0000 7f97d6a8c640 1 Processor -- start 2024-04-25T14:02:45.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.445+0000 7f97d6a8c640 1 -- start start 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d01e6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97d015f150 0x7f97d015de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97d006cfb0 con 0x7f97d006d4d0 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f97d006ccd0 con 0x7f97d015f150 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d6a8c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97d01479f0 con 0x7f97d0073c10 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56592/0 (socket says 172.21.15.150:56592) 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 -- 172.21.15.150:0/315391890 learned_addr learned my addr 172.21.15.150:0/315391890 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d5029640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97d015f150 0x7f97d015de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97cffff640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d01e6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 -- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 msgr2=0x7f97d01e6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d01e6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 -- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97d015f150 msgr2=0x7f97d015de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97d015f150 0x7f97d015de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97d4828640 1 -- 172.21.15.150:0/315391890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d01e7e50 con 0x7f97d006d4d0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.446+0000 7f97cffff640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d01e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97d4828640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c3c0 tx=0x7f97c000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0017070 con 0x7f97d006d4d0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97c00052f0 con 0x7f97d006d4d0 2024-04-25T14:02:45.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d01e7890 con 0x7f97d006d4d0 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c0003850 con 0x7f97d006d4d0 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.447+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d01445a0 con 0x7f97d006d4d0 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.448+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97d0076a40 con 0x7f97d006d4d0 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.449+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f97c001a070 con 0x7f97d006d4d0 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.449+0000 7f97cd7fa640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 0x7f97bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:45.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.449+0000 7f97cffff640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 0x7f97bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:45.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.449+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f97c00948b0 con 0x7f97d006d4d0 2024-04-25T14:02:45.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.450+0000 7f97cffff640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 0x7f97bc0769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f97c4034000 tx=0x7f97c402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:45.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.454+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f97d0074020 con 0x7f97d006d4d0 2024-04-25T14:02:45.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.664+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f97d0146f60 con 0x7f97d006d4d0 2024-04-25T14:02:45.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.665+0000 7f97cd7fa640 1 -- 172.21.15.150:0/315391890 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f97c005e6b0 con 0x7f97d006d4d0 2024-04-25T14:02:45.664 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T14:02:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 msgr2=0x7f97bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 0x7f97bc0769c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f97c4034000 tx=0x7f97c402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 msgr2=0x7f97d01f3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f97c000c3c0 tx=0x7f97c000c880 comp rx=0 tx=0).stop 2024-04-25T14:02:45.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 shutdown_connections 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97bc0744b0 0x7f97bc0769c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f97d015f150 0x7f97d015de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f97d0073c10 0x7f97d01e6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 --2- 172.21.15.150:0/315391890 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97d006d4d0 0x7f97d01f3d30 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 >> 172.21.15.150:0/315391890 conn(0x7f97d00d1b40 msgr2=0x7f97d01600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 shutdown_connections 2024-04-25T14:02:45.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45.667+0000 7f97d6a8c640 1 -- 172.21.15.150:0/315391890 wait complete. 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:02:45.677 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-25T14:02:45.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:45.678 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:45.693 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42734sec preferred_lft 42734sec 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::eb5b:7bab:e07b:2aac/64 scope link noprefixroute 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fe:50:c0:ba:66:ff brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.694 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::e835:8ff:febc:ce5a/64 scope link 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fe:50:c0:ba:66:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::fc50:c0ff:feba:66ff/64 scope link tentative 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:45.695 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:02:45.755 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-25T14:02:45.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:45.756 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-04-25T14:02:45.812 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-04-25T14:02:45.813 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:02:45.813 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:02:45.813 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:02:45.813 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:02:45.813 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:45.813 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:45.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:02:45.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:02:45.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:02:45.814 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:45.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:45.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:45.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:45.936 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:02:45.950 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T14:02:45.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:45.951 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list-id 2024-04-25T14:02:46.006 INFO:teuthology.orchestra.run.smithi150.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-25T14:02:46.006 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:46.006 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:46.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T14:02:46.006 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-25T14:02:46.006 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:46.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:46.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:46.120 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 3e:b1:51:74:b5:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T14:02:46.143 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.144 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::3cb1:51ff:fe74:b5dd/64 scope link 2024-04-25T14:02:46.144 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:46.145 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-25T14:02:46.208 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 3e:b1:51:74:b5:dd brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::3cb1:51ff:fe74:b5dd/64 scope link 2024-04-25T14:02:46.209 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:46.210 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-25T14:02:46.211 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:46.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:46.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.359 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:46.359 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:02:46.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.1 up 2024-04-25T14:02:46.359 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T14:02:46.359 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:02:46.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:46.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.434 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:46.434 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-25T14:02:46.448 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-25T14:02:46.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:46.449 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-25T14:02:46.503 INFO:tasks.cephfs.kernel_mount:mounting using device: 2@.cephfs=/ 2024-04-25T14:02:46.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:46.503 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 2@.cephfs=/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:46.596 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:46.596 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:46.609 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:46.609 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:46.609 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:02:46.609 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with new device syntax: 2@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:02:46.609 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,ms_mode=prefer-crc,key=2,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:02:46.611 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:02:46.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:46.611 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-25T14:02:46.635 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:46.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:46.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.731 INFO:teuthology.orchestra.run.smithi150.stdout:client4869 2024-04-25T14:02:46.732 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:46.732 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:46.748 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:46.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:46.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.844 INFO:teuthology.orchestra.run.smithi150.stdout:client4869 2024-04-25T14:02:46.845 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:46.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:46.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.900 INFO:teuthology.orchestra.run.smithi150.stdout:client4869 2024-04-25T14:02:46.901 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.2 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/77163e42-55ae-4cab-b506-bd8c531cc087.client4869/status') 2024-04-25T14:02:46.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:46.958 INFO:teuthology.orchestra.run.smithi150.stdout:instance: client.4869 (3)192.168.0.2:0/3997928789 2024-04-25T14:02:46.958 INFO:teuthology.orchestra.run.smithi150.stdout:blocklisted: false 2024-04-25T14:02:46.958 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records in 2024-04-25T14:02:46.958 INFO:teuthology.orchestra.run.smithi150.stderr:0+1 records out 2024-04-25T14:02:46.959 INFO:teuthology.orchestra.run.smithi150.stderr:69 bytes copied, 2.2468e-05 s, 3.1 MB/s 2024-04-25T14:02:46.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:46 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:46.966 INFO:tasks.kclient:client.1 config is {'syntax': 'v2'} 2024-04-25T14:02:46.966 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T14:02:46.966 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T14:02:46.966 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:02:47.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.074+0000 7fa088230640 1 Processor -- start 2024-04-25T14:02:47.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.074+0000 7fa088230640 1 -- start start 2024-04-25T14:02:47.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa088230640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa0800b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa088230640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0800b3c30 con 0x7fa08006f500 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa088230640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0800b3f10 con 0x7fa0800cfd10 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa088230640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0800b39e0 con 0x7fa0800d22e0 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa0867cd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa08006f500 0x7fa0800b1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41296/0 (socket says 172.21.15.150:41296) 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.075+0000 7fa0867cd640 1 -- 172.21.15.150:0/554583565 learned_addr learned my addr 172.21.15.150:0/554583565 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa0857cb640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa0800b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2051980296 0 0) 0x7fa0800b3c30 con 0x7fa08006f500 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa06c003520 con 0x7fa08006f500 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443018278 0 0) 0x7fa0800b3f10 con 0x7fa0800cfd10 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0800b3c30 con 0x7fa0800cfd10 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1752429237 0 0) 0x7fa06c003520 con 0x7fa08006f500 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 msgr2=0x7fa0800b4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:47.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa0800b4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 >> v1:172.21.15.187:6789/0 conn(0x7fa0800cfd10 legacy=0x7fa0800d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0800b41f0 con 0x7fa08006f500 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fa0780019f0 con 0x7fa08006f500 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 >> v1:172.21.15.150:6789/0 conn(0x7fa08006f500 legacy=0x7fa0800b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa06c003b70 0x7fa06c003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c004800 0x7fa06c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa06c0052c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa0800b39e0 con 0x7fa06c003b70 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa06c00bc20 con 0x7fa06c004800 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa06c00be20 con 0x7fa0800d22e0 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa085fcc640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa06c003b70 0x7fa06c003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa0857cb640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c004800 0x7fa06c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa0867cd640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa06c0052c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa085fcc640 1 -- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 msgr2=0x7fa06c0052c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:47.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa085fcc640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0800d22e0 0x7fa06c0052c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa085fcc640 1 -- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c004800 msgr2=0x7fa06c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.076+0000 7fa085fcc640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c004800 0x7fa06c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.077+0000 7fa085fcc640 1 -- 172.21.15.150:0/554583565 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa06c003520 con 0x7fa06c003b70 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.077+0000 7fa085fcc640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa06c003b70 0x7fa06c003fc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0018a0 tx=0x7fa06c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a4e77a7a1eba654 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.077+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c0014f0 con 0x7fa06c003b70 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c0052d0 con 0x7fa06c003b70 2024-04-25T14:02:47.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa084fca640 1 -- 172.21.15.150:0/554583565 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c02ac60 con 0x7fa06c003b70 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 -- 172.21.15.150:0/554583565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa06c003b70 msgr2=0x7fa06c003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 --2- 172.21.15.150:0/554583565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa06c003b70 0x7fa06c003fc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0018a0 tx=0x7fa06c005830 comp rx=0 tx=0).stop 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa0867cd640 1 -- 172.21.15.150:0/554583565 reap_dead start 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 -- 172.21.15.150:0/554583565 shutdown_connections 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 -- 172.21.15.150:0/554583565 >> 172.21.15.150:0/554583565 conn(0x7fa08009b4a0 msgr2=0x7fa08009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 -- 172.21.15.150:0/554583565 shutdown_connections 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.078+0000 7fa088230640 1 -- 172.21.15.150:0/554583565 wait complete. 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.080+0000 7fa088230640 1 Processor -- start 2024-04-25T14:02:47.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.080+0000 7fa088230640 1 -- start start 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.081+0000 7fa088230640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c003b70 0x7fa0800c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.081+0000 7fa088230640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08015eec0 0x7fa080070e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.081+0000 7fa088230640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa088230640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa08006f910 con 0x7fa08015f2d0 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa088230640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0800b47b0 con 0x7fa06c003b70 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa088230640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0800b4cd0 con 0x7fa08015eec0 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:56606/0 (socket says 172.21.15.150:56606) 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 -- 172.21.15.150:0/2242696738 learned_addr learned my addr 172.21.15.150:0/2242696738 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa085fcc640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c003b70 0x7fa0800c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0857cb640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08015eec0 0x7fa080070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 -- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08015eec0 msgr2=0x7fa080070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08015eec0 0x7fa080070e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 -- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c003b70 msgr2=0x7fa0800c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c003b70 0x7fa0800c7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 -- 172.21.15.150:0/2242696738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0800c9100 con 0x7fa08015f2d0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0867cd640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0800d3970 tx=0x7fa078007b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07800a990 con 0x7fa08015f2d0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa078018070 con 0x7fa08015f2d0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0801fa0e0 con 0x7fa08015f2d0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07800ad70 con 0x7fa08015f2d0 2024-04-25T14:02:47.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.082+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0800c8e20 con 0x7fa08015f2d0 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.083+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0800c88b0 con 0x7fa08015f2d0 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.084+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa07801d460 con 0x7fa08015f2d0 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.084+0000 7fa0767fc640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 0x7fa0640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.085+0000 7fa085fcc640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 0x7fa0640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.085+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7fa0780957a0 con 0x7fa08015f2d0 2024-04-25T14:02:47.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.085+0000 7fa085fcc640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 0x7fa0640769f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa07c032000 tx=0x7fa07c009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:47.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.089+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa07805fbf0 con 0x7fa08015f2d0 2024-04-25T14:02:47.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.300+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa080069c80 con 0x7fa08015f2d0 2024-04-25T14:02:47.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.301+0000 7fa0767fc640 1 -- 172.21.15.150:0/2242696738 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fa07805f4f0 con 0x7fa08015f2d0 2024-04-25T14:02:47.300 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T14:02:47.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 msgr2=0x7fa0640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 0x7fa0640769f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa07c032000 tx=0x7fa07c009030 comp rx=0 tx=0).stop 2024-04-25T14:02:47.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 msgr2=0x7fa080071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:47.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa0800d3970 tx=0x7fa078007b10 comp rx=0 tx=0).stop 2024-04-25T14:02:47.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 shutdown_connections 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0640744e0 0x7fa0640769f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa08015f2d0 0x7fa080071370 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa08015eec0 0x7fa080070e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 --2- 172.21.15.150:0/2242696738 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa06c003b70 0x7fa0800c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 >> 172.21.15.150:0/2242696738 conn(0x7fa08009b4a0 msgr2=0x7fa08015fe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 shutdown_connections 2024-04-25T14:02:47.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:47.303+0000 7fa088230640 1 -- 172.21.15.150:0/2242696738 wait complete. 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T14:02:47.313 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:02:47.314 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-25T14:02:47.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:47.314 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2024-04-25T14:02:47.332 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:47.332 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f0 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f1 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:02:47.333 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:02:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42711sec preferred_lft 42711sec 2024-04-25T14:02:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ce79:901d:95e6:7ec7/64 scope link noprefixroute 2024-04-25T14:02:47.334 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:47.334 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add name ceph-brx type bridge 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr flush dev ceph-brx 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set ceph-brx up 2024-04-25T14:02:47.334 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T14:02:47.335 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:47.428 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:47.519 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:47.526 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:47.526 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:02:47.550 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-25T14:02:47.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:47.551 DEBUG:teuthology.orchestra.run.smithi187:> route 2024-04-25T14:02:47.618 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2024-04-25T14:02:47.618 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:02:47.618 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:02:47.618 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:02:47.618 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:02:47.619 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:47.706 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:47 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:47.906 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:47 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:47.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:47.912 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:02:47.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:47.927 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2024-04-25T14:02:47.981 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:47.981 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:47.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T14:02:47.982 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-25T14:02:47.982 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:48.083 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:48.098 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.106 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-25T14:02:48.107 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:48.107 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:48.107 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-25T14:02:48.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T14:02:48.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-25T14:02:48.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-25T14:02:48.109 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-25T14:02:48.109 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:48.163 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:48.231 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.233 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:48.234 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:48.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.0 up 2024-04-25T14:02:48.234 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T14:02:48.234 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:48.281 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:48.302 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:48.309 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-25T14:02:48.323 INFO:teuthology.orchestra.run.smithi187.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-25T14:02:48.324 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:48.324 DEBUG:teuthology.orchestra.run.smithi187:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-25T14:02:48.378 INFO:tasks.cephfs.kernel_mount:mounting using device: 1@.cephfs=/ 2024-04-25T14:02:48.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:48.378 DEBUG:teuthology.orchestra.run.smithi187:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 1@.cephfs=/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:48.474 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:48.475 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:48.573 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:48.573 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:48.573 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:02:48.573 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: trying mount with new device syntax: 1@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:02:48.573 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,ms_mode=prefer-crc,key=1,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:02:48.574 INFO:teuthology.orchestra.run.smithi187.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-04-25T14:02:48.574 INFO:teuthology.orchestra.run.smithi187.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:02:48.574 INFO:teuthology.orchestra.run.smithi187.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:02:48.574 INFO:teuthology.orchestra.run.smithi187.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:02:48.574 INFO:teuthology.orchestra.run.smithi187.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:02:48.576 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:02:48.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:48.576 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-25T14:02:48.599 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:48.688 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:48.689 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.695 INFO:teuthology.orchestra.run.smithi187.stdout:client4878 2024-04-25T14:02:48.696 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:02:48.696 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:48.712 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:48.800 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:48.801 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.807 INFO:teuthology.orchestra.run.smithi187.stdout:client4878 2024-04-25T14:02:48.808 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:48.855 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:48.857 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.862 INFO:teuthology.orchestra.run.smithi187.stdout:client4878 2024-04-25T14:02:48.863 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.1 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/77163e42-55ae-4cab-b506-bd8c531cc087.client4878/status') 2024-04-25T14:02:48.910 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:48.919 INFO:teuthology.orchestra.run.smithi187.stdout:instance: client.4878 (3)172.21.15.187:0/2207478570 2024-04-25T14:02:48.920 INFO:teuthology.orchestra.run.smithi187.stdout:blocklisted: false 2024-04-25T14:02:48.920 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2024-04-25T14:02:48.920 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2024-04-25T14:02:48.920 INFO:teuthology.orchestra.run.smithi187.stderr:71 bytes copied, 2.255e-05 s, 3.1 MB/s 2024-04-25T14:02:48.920 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:48.927 INFO:tasks.kclient:client.3 config is {'syntax': 'v2'} 2024-04-25T14:02:48.927 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T14:02:48.927 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T14:02:48.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:02:49.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.049+0000 7f20de3ce640 1 Processor -- start 2024-04-25T14:02:49.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.050+0000 7f20de3ce640 1 -- start start 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20de3ce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20d815d110 0x7f20d80ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20de3ce640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f20d8160c10 0x7f20d8161020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20de3ce640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d8161560 con 0x7f20d80b4d50 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20de3ce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f20d806e9e0 con 0x7f20d815d110 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20de3ce640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f20d80b26b0 con 0x7f20d8160c10 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20d815d110 0x7f20d80ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f20d815d110 0x7f20d80ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44652/0 (socket says 172.21.15.150:44652) 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 -- 172.21.15.150:0/604044364 learned_addr learned my addr 172.21.15.150:0/604044364 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20dc96b640 1 --2- 172.21.15.150:0/604044364 >> v2:172.21.15.187:3300/0 conn(0x7f20d8160c10 0x7f20d8161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 -- 172.21.15.150:0/604044364 >> v2:172.21.15.187:3300/0 conn(0x7f20d8160c10 msgr2=0x7f20d8161020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:49.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 --2- 172.21.15.150:0/604044364 >> v2:172.21.15.187:3300/0 conn(0x7f20d8160c10 0x7f20d8161020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.051+0000 7f20d77fe640 1 -- 172.21.15.150:0/604044364 >> v1:172.21.15.187:6789/0 conn(0x7f20d80b4d50 legacy=0x7f20d80b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:49.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.052+0000 7f20d77fe640 1 -- 172.21.15.150:0/604044364 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d815d520 con 0x7f20d815d110 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.052+0000 7f20d77fe640 1 --2- 172.21.15.150:0/604044364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d815d110 0x7f20d80ccf20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f20d80b57d0 tx=0x7f20cc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f895339a34a87d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.052+0000 7f20d6ffd640 1 -- 172.21.15.150:0/604044364 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc037070 con 0x7f20d815d110 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.052+0000 7f20d6ffd640 1 -- 172.21.15.150:0/604044364 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20cc032440 con 0x7f20d815d110 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.052+0000 7f20d6ffd640 1 -- 172.21.15.150:0/604044364 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20cc032810 con 0x7f20d815d110 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 -- 172.21.15.150:0/604044364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d815d110 msgr2=0x7f20d80ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 --2- 172.21.15.150:0/604044364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d815d110 0x7f20d80ccf20 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f20d80b57d0 tx=0x7f20cc02a230 comp rx=0 tx=0).stop 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 -- 172.21.15.150:0/604044364 shutdown_connections 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 --2- 172.21.15.150:0/604044364 >> v2:172.21.15.187:3300/0 conn(0x7f20d8160c10 0x7f20d8161020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 --2- 172.21.15.150:0/604044364 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d815d110 0x7f20d80ccf20 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 -- 172.21.15.150:0/604044364 >> 172.21.15.150:0/604044364 conn(0x7f20d809b4a0 msgr2=0x7f20d809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 -- 172.21.15.150:0/604044364 shutdown_connections 2024-04-25T14:02:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.053+0000 7f20de3ce640 1 -- 172.21.15.150:0/604044364 wait complete. 2024-04-25T14:02:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.055+0000 7f20de3ce640 1 Processor -- start 2024-04-25T14:02:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.055+0000 7f20de3ce640 1 -- start start 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.056+0000 7f20de3ce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d815d110 0x7f20d81f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d8160c10 0x7f20d80c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f20d806e670 con 0x7f20d80b4d50 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f20d806e390 con 0x7f20d815d110 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f20d80cd460 con 0x7f20d8160c10 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44662/0 (socket says 172.21.15.150:44662) 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 -- 172.21.15.150:0/1126803159 learned_addr learned my addr 172.21.15.150:0/1126803159 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d77fe640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d815d110 0x7f20d81f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20dc96b640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d8160c10 0x7f20d80c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 -- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d8160c10 msgr2=0x7f20d80c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d8160c10 0x7f20d80c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 -- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d815d110 msgr2=0x7f20d81f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d815d110 0x7f20d81f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 -- 172.21.15.150:0/1126803159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d8156a00 con 0x7f20d80b4d50 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d7fff640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20c800c390 tx=0x7f20c800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8011790 con 0x7f20d80b4d50 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c8017070 con 0x7f20d80b4d50 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d8156ca0 con 0x7f20d80b4d50 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.057+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8011b70 con 0x7f20d80b4d50 2024-04-25T14:02:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.058+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d8157260 con 0x7f20d80b4d50 2024-04-25T14:02:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.058+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d8076a40 con 0x7f20d80b4d50 2024-04-25T14:02:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.059+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f20c8011d10 con 0x7f20d80b4d50 2024-04-25T14:02:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.060+0000 7f20d4ff9640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.060+0000 7f20d77fe640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.060+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6532+0+0 (secure 0 0 0) 0x7f20c80951e0 con 0x7f20d80b4d50 2024-04-25T14:02:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.060+0000 7f20d77fe640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 0x7f20c40769c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20cc034000 tx=0x7f20cc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:49.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.064+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20c805efe0 con 0x7f20d80b4d50 2024-04-25T14:02:49.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.280+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f20d80b2350 con 0x7f20d80b4d50 2024-04-25T14:02:49.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.281+0000 7f20d4ff9640 1 -- 172.21.15.150:0/1126803159 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f20c80632e0 con 0x7f20d80b4d50 2024-04-25T14:02:49.280 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-25T14:02:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 msgr2=0x7f20c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:49.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 0x7f20c40769c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f20cc034000 tx=0x7f20cc02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 msgr2=0x7f20d8157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:49.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f20c800c390 tx=0x7f20c800c850 comp rx=0 tx=0).stop 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 shutdown_connections 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f20c40744b0 0x7f20c40769c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f20d8160c10 0x7f20d80c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f20d815d110 0x7f20d81f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 --2- 172.21.15.150:0/1126803159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f20d80b4d50 0x7f20d8157780 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 >> 172.21.15.150:0/1126803159 conn(0x7f20d809b4a0 msgr2=0x7f20d80ce150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 shutdown_connections 2024-04-25T14:02:49.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:49.283+0000 7f20de3ce640 1 -- 172.21.15.150:0/1126803159 wait complete. 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi187.front.sepia.ceph.com 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T14:02:49.294 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:02:49.295 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-25T14:02:49.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.295 DEBUG:teuthology.orchestra.run.smithi187:> ip addr 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:78 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f0 2024-04-25T14:02:49.312 INFO:teuthology.orchestra.run.smithi187.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:d9:36:79 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: altname enp7s0f1 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b0 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 0c:c4:7a:88:77:b1 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: inet 172.21.15.187/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft 42709sec preferred_lft 42709sec 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ce79:901d:95e6:7ec7/64 scope link noprefixroute 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether ee:16:6e:e3:1f:39 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.313 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::8cbc:b7ff:fe67:245a/64 scope link 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run.smithi187.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether ee:16:6e:e3:1f:39 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::ec16:6eff:fee3:1f39/64 scope link tentative 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.314 DEBUG:teuthology.orchestra.run.smithi187:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:02:49.376 INFO:teuthology.orchestra.run.smithi187.stdout:1 2024-04-25T14:02:49.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.377 DEBUG:teuthology.orchestra.run.smithi187:> route 2024-04-25T14:02:49.435 INFO:teuthology.orchestra.run.smithi187.stdout:Kernel IP routing table 2024-04-25T14:02:49.435 INFO:teuthology.orchestra.run.smithi187.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:02:49.435 INFO:teuthology.orchestra.run.smithi187.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:02:49.435 INFO:teuthology.orchestra.run.smithi187.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:02:49.435 INFO:teuthology.orchestra.run.smithi187.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:02:49.436 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:49.526 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:49.558 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:49.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.565 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:02:49.580 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T14:02:49.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.581 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list-id 2024-04-25T14:02:49.635 INFO:teuthology.orchestra.run.smithi187.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-25T14:02:49.636 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:49.636 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:49.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T14:02:49.636 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-25T14:02:49.636 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:49.723 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:49.744 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:49.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.751 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T14:02:49.774 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:49.774 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 5e:8b:38:ca:a6:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::5c8b:38ff:feca:a605/64 scope link tentative 2024-04-25T14:02:49.775 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:49.776 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 ::1/128 scope host 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: link/ether 5e:8b:38:ca:a6:05 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-25T14:02:49.841 INFO:teuthology.orchestra.run.smithi187.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-25T14:02:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: inet6 fe80::5c8b:38ff:feca:a605/64 scope link tentative 2024-04-25T14:02:49.842 INFO:teuthology.orchestra.run.smithi187.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:02:49.843 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-25T14:02:49.843 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:49.843 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:49.843 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-25T14:02:49.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-25T14:02:49.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-25T14:02:49.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-25T14:02:49.844 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-25T14:02:49.844 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:49.934 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:49.986 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:49 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:49.992 DEBUG:teuthology.orchestra.run.smithi187:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:02:49.993 DEBUG:teuthology.orchestra.run.smithi187:> set -e 2024-04-25T14:02:49.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set brx.1 up 2024-04-25T14:02:49.993 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link set dev brx.1 master ceph-brx 2024-04-25T14:02:49.993 DEBUG:teuthology.orchestra.run.smithi187:> ') 2024-04-25T14:02:50.040 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:50.061 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:50.067 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:50.067 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-25T14:02:50.082 INFO:teuthology.orchestra.run.smithi187.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-25T14:02:50.082 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:02:50.082 DEBUG:teuthology.orchestra.run.smithi187:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-25T14:02:50.138 INFO:tasks.cephfs.kernel_mount:mounting using device: 3@.cephfs=/ 2024-04-25T14:02:50.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:50.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 3@.cephfs=/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:50.233 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:50.233 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:50.241 INFO:teuthology.orchestra.run.smithi187.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:02:50.242 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:02:50.242 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:02:50.242 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: trying mount with new device syntax: 3@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:02:50.242 INFO:teuthology.orchestra.run.smithi187.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,ms_mode=prefer-crc,key=3,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:02:50.244 INFO:teuthology.orchestra.run.smithi187.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-04-25T14:02:50.244 INFO:teuthology.orchestra.run.smithi187.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:02:50.244 INFO:teuthology.orchestra.run.smithi187.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:02:50.244 INFO:teuthology.orchestra.run.smithi187.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:02:50.244 INFO:teuthology.orchestra.run.smithi187.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:02:50.245 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:02:50.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:02:50.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-25T14:02:50.269 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:50.358 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:50.360 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:50.368 INFO:teuthology.orchestra.run.smithi187.stdout:client4887 2024-04-25T14:02:50.368 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:02:50.368 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:50.384 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:50.474 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:50.476 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:50.480 INFO:teuthology.orchestra.run.smithi187.stdout:client4887 2024-04-25T14:02:50.480 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:02:50.529 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:02:50.531 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:50.536 INFO:teuthology.orchestra.run.smithi187.stdout:client4887 2024-04-25T14:02:50.536 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest/mnt.3 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/77163e42-55ae-4cab-b506-bd8c531cc087.client4887/status') 2024-04-25T14:02:50.587 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:02:50.596 INFO:teuthology.orchestra.run.smithi187.stdout:instance: client.4887 (3)172.21.15.187:0/1021105199 2024-04-25T14:02:50.597 INFO:teuthology.orchestra.run.smithi187.stdout:blocklisted: false 2024-04-25T14:02:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records in 2024-04-25T14:02:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:0+1 records out 2024-04-25T14:02:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:71 bytes copied, 2.3083e-05 s, 3.1 MB/s 2024-04-25T14:02:50.597 INFO:teuthology.orchestra.run.smithi187.stderr:2024-04-25T14:02:50 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:02:50.603 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-25T14:02:50.621 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:50.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.744+0000 7f2655196640 1 Processor -- start 2024-04-25T14:02:50.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.744+0000 7f2655196640 1 -- start start 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.745+0000 7f2655196640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f2655196640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26500b4750 con 0x7f265006f780 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f2655196640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26500b4a30 con 0x7f26500b5f00 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f2655196640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f265015e310 con 0x7f265015df00 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264e59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f265006f780 0x7f26500b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55170/0 (socket says 172.21.15.150:55170) 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264e59c640 1 -- 172.21.15.150:0/3708604256 learned_addr learned my addr 172.21.15.150:0/3708604256 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601937903 0 0) 0x7f26500b4750 con 0x7f265006f780 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264ed9d640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2638003520 con 0x7f265006f780 2024-04-25T14:02:50.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 829347704 0 0) 0x7f2638003520 con 0x7f265006f780 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 msgr2=0x7f265006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265006f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 >> v1:172.21.15.187:6789/0 conn(0x7f26500b5f00 legacy=0x7f26500ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26500b4d10 con 0x7f265006f780 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f2644001670 con 0x7f265006f780 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 >> v1:172.21.15.150:6789/0 conn(0x7f265006f780 legacy=0x7f26500b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f26380039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2638003f60 0x7f2638004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2638004d40 0x7f2638005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f265015e310 con 0x7f2638004d40 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f263800bc00 con 0x7f2638003f60 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f263800e010 con 0x7f265015df00 2024-04-25T14:02:50.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264ed9d640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f26380039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264e59c640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2638003f60 0x7f2638004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.746+0000 7f264f59e640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2638004d40 0x7f2638005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 -- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2638003f60 msgr2=0x7f2638004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2638003f60 0x7f2638004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 -- 172.21.15.150:0/3708604256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2638004d40 msgr2=0x7f2638005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2638004d40 0x7f2638005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 -- 172.21.15.150:0/3708604256 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2638003520 con 0x7f265015df00 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.747+0000 7f264ed9d640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f26380039e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2640009ac0 tx=0x7f2638005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68489bcbccb7de89 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640036070 con 0x7f265015df00 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2640031400 con 0x7f265015df00 2024-04-25T14:02:50.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f264dd9b640 1 -- 172.21.15.150:0/3708604256 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640009690 con 0x7f265015df00 2024-04-25T14:02:50.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 -- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 msgr2=0x7f26380039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 --2- 172.21.15.150:0/3708604256 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f26380039e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2640009ac0 tx=0x7f2638005830 comp rx=0 tx=0).stop 2024-04-25T14:02:50.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f264f59e640 1 -- 172.21.15.150:0/3708604256 reap_dead start 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 -- 172.21.15.150:0/3708604256 shutdown_connections 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 -- 172.21.15.150:0/3708604256 >> 172.21.15.150:0/3708604256 conn(0x7f26500d1ae0 msgr2=0x7f26500d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 -- 172.21.15.150:0/3708604256 shutdown_connections 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.748+0000 7f2655196640 1 -- 172.21.15.150:0/3708604256 wait complete. 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.750+0000 7f2655196640 1 Processor -- start 2024-04-25T14:02:50.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.751+0000 7f2655196640 1 -- start start 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 0x7f2650142db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f26500aeef0 con 0x7f2650156d00 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f26500b5360 con 0x7f26501429a0 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f2655196640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f26500af070 con 0x7f265015df00 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44680/0 (socket says 172.21.15.150:44680) 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 -- 172.21.15.150:0/986799312 learned_addr learned my addr 172.21.15.150:0/986799312 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:50.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264e59c640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 0x7f2650142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264ed9d640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 -- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 msgr2=0x7f265013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265013fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 -- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 msgr2=0x7f2650142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 0x7f2650142db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 -- 172.21.15.150:0/986799312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26501432f0 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264e59c640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 0x7f2650142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.752+0000 7f264f59e640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f26500745d0 tx=0x7f264800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.753+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264801a070 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.753+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2648006d60 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.753+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2650151820 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.753+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2648003b90 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.753+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2650151de0 con 0x7f2650156d00 2024-04-25T14:02:50.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.754+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26500ae7e0 con 0x7f2650156d00 2024-04-25T14:02:50.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.754+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f264801d070 con 0x7f2650156d00 2024-04-25T14:02:50.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.755+0000 7f26377fe640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 0x7f263c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:50.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.755+0000 7f264ed9d640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 0x7f263c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:50.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.755+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2648094b40 con 0x7f2650156d00 2024-04-25T14:02:50.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.756+0000 7f264ed9d640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 0x7f263c0769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2640033000 tx=0x7f2640008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:50.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.760+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f26500ae7e0 con 0x7f2650156d00 2024-04-25T14:02:50.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.936+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2650076a40 con 0x7f2650156d00 2024-04-25T14:02:50.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.938+0000 7f26377fe640 1 -- 172.21.15.150:0/986799312 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7f26501432f0 con 0x7f2650156d00 2024-04-25T14:02:50.937 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:50.937 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 msgr2=0x7f263c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 0x7f263c0769c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2640033000 tx=0x7f2640008030 comp rx=0 tx=0).stop 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 msgr2=0x7f2650157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f26500745d0 tx=0x7f264800fd40 comp rx=0 tx=0).stop 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 shutdown_connections 2024-04-25T14:02:50.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f263c0744b0 0x7f263c0769c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2650156d00 0x7f2650157110 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f26501429a0 0x7f2650142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 --2- 172.21.15.150:0/986799312 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f265015df00 0x7f265013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 >> 172.21.15.150:0/986799312 conn(0x7f26500d1ae0 msgr2=0x7f265015a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 shutdown_connections 2024-04-25T14:02:50.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:50.940+0000 7f2655196640 1 -- 172.21.15.150:0/986799312 wait complete. 2024-04-25T14:02:50.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:50.951 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:51.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.062+0000 7f398e492640 1 Processor -- start 2024-04-25T14:02:51.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.063+0000 7f398e492640 1 -- start start 2024-04-25T14:02:51.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398e492640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f398815a850 0x7f398809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398e492640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398e492640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f398815ac60 con 0x7f3988160770 2024-04-25T14:02:51.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398e492640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f398809bdf0 con 0x7f398815a850 2024-04-25T14:02:51.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398e492640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3988160ce0 con 0x7f398809b9e0 2024-04-25T14:02:51.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398ca2f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f39877fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f398815a850 0x7f398809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398ca2f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:44520/0 (socket says 172.21.15.150:44520) 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f39877fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f398815a850 0x7f398809b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:36854/0 (socket says 172.21.15.150:36854) 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f398ca2f640 1 -- 172.21.15.150:0/2243720786 learned_addr learned my addr 172.21.15.150:0/2243720786 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.064+0000 7f3987fff640 1 --1- 172.21.15.150:0/2243720786 >> v1:172.21.15.187:6789/0 conn(0x7f3988160770 0x7f398806f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:35992/0 (socket says 172.21.15.150:35992) 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f39877fe640 1 -- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 msgr2=0x7f39880b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f39877fe640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f39877fe640 1 -- 172.21.15.150:0/2243720786 >> v1:172.21.15.187:6789/0 conn(0x7f3988160770 legacy=0x7f398806f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f39877fe640 1 -- 172.21.15.150:0/2243720786 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3988160fc0 con 0x7f398815a850 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f398ca2f640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f39877fe640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f398815a850 0x7f398809b4a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f397c0097a0 tx=0x7f397c02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50bdf7328c69a584 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.065+0000 7f3986ffd640 1 -- 172.21.15.150:0/2243720786 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c037070 con 0x7f398815a850 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f3986ffd640 1 -- 172.21.15.150:0/2243720786 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f397c032440 con 0x7f398815a850 2024-04-25T14:02:51.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f3986ffd640 1 -- 172.21.15.150:0/2243720786 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f397c032810 con 0x7f398815a850 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 -- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f398815a850 msgr2=0x7f398809b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f398815a850 0x7f398809b4a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f397c0097a0 tx=0x7f397c02a180 comp rx=0 tx=0).stop 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 -- 172.21.15.150:0/2243720786 shutdown_connections 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39880b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 --2- 172.21.15.150:0/2243720786 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f398815a850 0x7f398809b4a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 -- 172.21.15.150:0/2243720786 >> 172.21.15.150:0/2243720786 conn(0x7f39880b7680 msgr2=0x7f39880b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 -- 172.21.15.150:0/2243720786 shutdown_connections 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.066+0000 7f398e492640 1 -- 172.21.15.150:0/2243720786 wait complete. 2024-04-25T14:02:51.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.068+0000 7f398e492640 1 Processor -- start 2024-04-25T14:02:51.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.068+0000 7f398e492640 1 -- start start 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39881f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f39880b35b0 con 0x7f398815a850 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3988161580 con 0x7f3988160770 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.069+0000 7f398e492640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f39880b3730 con 0x7f398809b9e0 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f39877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:36868/0 (socket says 172.21.15.150:36868) 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 -- 172.21.15.150:0/3629188041 learned_addr learned my addr 172.21.15.150:0/3629188041 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f39877fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44686/0 (socket says 172.21.15.150:44686) 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f3987fff640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39881f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.071 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 -- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 msgr2=0x7f39881f6320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39881f6320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 -- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 msgr2=0x7f398809d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 -- 172.21.15.150:0/3629188041 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398809df90 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f39877fe640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398ca2f640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3980009e70 tx=0x7f398000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398000cbe0 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f398000cd80 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398809e270 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3980003bb0 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.070+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f398809e830 con 0x7f3988160770 2024-04-25T14:02:51.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.071+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398809dcf0 con 0x7f3988160770 2024-04-25T14:02:51.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.072+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f398001b030 con 0x7f3988160770 2024-04-25T14:02:51.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.072+0000 7f3984ff9640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 0x7f39740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.073+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3980095760 con 0x7f3988160770 2024-04-25T14:02:51.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.073+0000 7f3987fff640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 0x7f39740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.073+0000 7f3987fff640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 0x7f39740769f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3978007fd0 tx=0x7f3978007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.077+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f398809dcf0 con 0x7f3988160770 2024-04-25T14:02:51.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.250+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3988076a40 con 0x7f3988160770 2024-04-25T14:02:51.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.251+0000 7f3984ff9640 1 -- 172.21.15.150:0/3629188041 <== mon.1 v2:172.21.15.187:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f3980022020 con 0x7f3988160770 2024-04-25T14:02:51.250 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 msgr2=0x7f39740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 0x7f39740769f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3978007fd0 tx=0x7f3978007e80 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 msgr2=0x7f3988162850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3980009e70 tx=0x7f398000c490 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 shutdown_connections 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f39740744e0 0x7f39740769f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3988160770 0x7f3988162850 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f398815a850 0x7f398809d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 --2- 172.21.15.150:0/3629188041 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f398809b9e0 0x7f39881f6320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 >> 172.21.15.150:0/3629188041 conn(0x7f39880b7680 msgr2=0x7f39880684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 shutdown_connections 2024-04-25T14:02:51.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.253+0000 7f398e492640 1 -- 172.21.15.150:0/3629188041 wait complete. 2024-04-25T14:02:51.264 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:51.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.378+0000 7f3f3819d640 1 Processor -- start 2024-04-25T14:02:51.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.379+0000 7f3f3819d640 1 -- start start 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3819d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f300d3290 0x7f3f30160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3819d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3f300735b0 0x7f3f300ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3819d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f30164e10 con 0x7f3f300cf330 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3819d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3f30160770 con 0x7f3f300d3290 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3819d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3f300b6c70 con 0x7f3f300735b0 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f300d3290 0x7f3f30160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3f300d3290 0x7f3f30160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44698/0 (socket says 172.21.15.150:44698) 2024-04-25T14:02:51.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 -- 172.21.15.150:0/3272497303 learned_addr learned my addr 172.21.15.150:0/3272497303 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:51.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f3673a640 1 --2- 172.21.15.150:0/3272497303 >> v2:172.21.15.187:3300/0 conn(0x7f3f300735b0 0x7f3f300ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 -- 172.21.15.150:0/3272497303 >> v2:172.21.15.187:3300/0 conn(0x7f3f300735b0 msgr2=0x7f3f300ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:51.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 --2- 172.21.15.150:0/3272497303 >> v2:172.21.15.187:3300/0 conn(0x7f3f300735b0 0x7f3f300ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 -- 172.21.15.150:0/3272497303 >> v1:172.21.15.187:6789/0 conn(0x7f3f300cf330 legacy=0x7f3f30072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.381 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.380+0000 7f3f35f39640 1 -- 172.21.15.150:0/3272497303 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f30164050 con 0x7f3f300d3290 2024-04-25T14:02:51.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.381+0000 7f3f35f39640 1 --2- 172.21.15.150:0/3272497303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300d3290 0x7f3f30160230 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f300d3980 tx=0x7f3f2c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=167b648e242013c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.381+0000 7f3f34f37640 1 -- 172.21.15.150:0/3272497303 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c036070 con 0x7f3f300d3290 2024-04-25T14:02:51.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.381+0000 7f3f34f37640 1 -- 172.21.15.150:0/3272497303 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c031600 con 0x7f3f300d3290 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.381+0000 7f3f34f37640 1 -- 172.21.15.150:0/3272497303 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c0319d0 con 0x7f3f300d3290 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 -- 172.21.15.150:0/3272497303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300d3290 msgr2=0x7f3f30160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 --2- 172.21.15.150:0/3272497303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300d3290 0x7f3f30160230 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f300d3980 tx=0x7f3f2c02a6a0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 -- 172.21.15.150:0/3272497303 shutdown_connections 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 --2- 172.21.15.150:0/3272497303 >> v2:172.21.15.187:3300/0 conn(0x7f3f300735b0 0x7f3f300ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 --2- 172.21.15.150:0/3272497303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300d3290 0x7f3f30160230 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 -- 172.21.15.150:0/3272497303 >> 172.21.15.150:0/3272497303 conn(0x7f3f300d39b0 msgr2=0x7f3f300d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 -- 172.21.15.150:0/3272497303 shutdown_connections 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.382+0000 7f3f3819d640 1 -- 172.21.15.150:0/3272497303 wait complete. 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.384+0000 7f3f3819d640 1 Processor -- start 2024-04-25T14:02:51.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.384+0000 7f3f3819d640 1 -- start start 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f300b3430 0x7f3f300b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f3006df10 0x7f3f3006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3f301648f0 con 0x7f3f300cf330 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3f30164610 con 0x7f3f300b3430 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3819d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3f300b6800 con 0x7f3f3006df10 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f35f39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f35f39640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44704/0 (socket says 172.21.15.150:44704) 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f35f39640 1 -- 172.21.15.150:0/4208601436 learned_addr learned my addr 172.21.15.150:0/4208601436 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:51.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f35738640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f300b3430 0x7f3f300b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.385+0000 7f3f3673a640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f3006df10 0x7f3f3006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 -- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f3006df10 msgr2=0x7f3f3006e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f3006df10 0x7f3f3006e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 -- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f300b3430 msgr2=0x7f3f300b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f300b3430 0x7f3f300b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 -- 172.21.15.150:0/4208601436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3006e860 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f35f39640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009f40 tx=0x7f3f2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c03f020 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f2c037040 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f300b41d0 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f2c036070 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.386+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f300b4790 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.387+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f30071170 con 0x7f3f300cf330 2024-04-25T14:02:51.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.388+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3f2c042020 con 0x7f3f300cf330 2024-04-25T14:02:51.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.388+0000 7f3f267fc640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 0x7f3f140769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.388+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f3f2c0b59a0 con 0x7f3f300cf330 2024-04-25T14:02:51.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.388+0000 7f3f35738640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 0x7f3f140769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.389+0000 7f3f35738640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 0x7f3f140769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3f300d1ee0 tx=0x7f3f20009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.393+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f30071170 con 0x7f3f300cf330 2024-04-25T14:02:51.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.569+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3f30076a40 con 0x7f3f300cf330 2024-04-25T14:02:51.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.569+0000 7f3f267fc640 1 -- 172.21.15.150:0/4208601436 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f3f2c049030 con 0x7f3f300cf330 2024-04-25T14:02:51.569 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 msgr2=0x7f3f140769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 0x7f3f140769c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3f300d1ee0 tx=0x7f3f20009040 comp rx=0 tx=0).stop 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 msgr2=0x7f3f3015cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3f2c009f40 tx=0x7f3f2c004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 shutdown_connections 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3f140744b0 0x7f3f140769c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3f3006df10 0x7f3f3006e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3f300b3430 0x7f3f300b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 --2- 172.21.15.150:0/4208601436 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3f300cf330 0x7f3f3015cc40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.571+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 >> 172.21.15.150:0/4208601436 conn(0x7f3f300d39b0 msgr2=0x7f3f3015a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.572+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 shutdown_connections 2024-04-25T14:02:51.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.572+0000 7f3f3819d640 1 -- 172.21.15.150:0/4208601436 wait complete. 2024-04-25T14:02:51.582 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:51.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.694+0000 7fdfb179d640 1 Processor -- start 2024-04-25T14:02:51.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.694+0000 7fdfb179d640 1 -- start start 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb179d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac0b9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb179d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 0x7fdfac160070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb179d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfac164880 con 0x7fdfac0d0970 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb179d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdfac1605b0 con 0x7fdfac0b46d0 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb179d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfac06cc60 con 0x7fdfac0b4c20 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb0f9c640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fdfac0d0970 0x7fdfac0d0d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:36000/0 (socket says 172.21.15.150:36000) 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfb0f9c640 1 -- 172.21.15.150:0/554532577 learned_addr learned my addr 172.21.15.150:0/554532577 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfabfff640 1 --2- 172.21.15.150:0/554532577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac0b9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfab7fe640 1 --2- 172.21.15.150:0/554532577 >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 0x7fdfac160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfaaffd640 1 -- 172.21.15.150:0/554532577 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765709839 0 0) 0x7fdfac164880 con 0x7fdfac0d0970 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfaaffd640 1 -- 172.21.15.150:0/554532577 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf94003520 con 0x7fdfac0d0970 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfabfff640 1 -- 172.21.15.150:0/554532577 >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 msgr2=0x7fdfac160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfabfff640 1 --2- 172.21.15.150:0/554532577 >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 0x7fdfac160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfabfff640 1 -- 172.21.15.150:0/554532577 >> v1:172.21.15.187:6789/0 conn(0x7fdfac0d0970 legacy=0x7fdfac0d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfabfff640 1 -- 172.21.15.150:0/554532577 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac164b60 con 0x7fdfac0b4c20 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.696+0000 7fdfab7fe640 1 --2- 172.21.15.150:0/554532577 >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 0x7fdfac160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:51.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.697+0000 7fdfabfff640 1 --2- 172.21.15.150:0/554532577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac0b9080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0097a0 tx=0x7fdf9c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=993f47212db31cec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.697+0000 7fdfaaffd640 1 -- 172.21.15.150:0/554532577 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c037070 con 0x7fdfac0b4c20 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.697+0000 7fdfaaffd640 1 -- 172.21.15.150:0/554532577 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c032440 con 0x7fdfac0b4c20 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.697+0000 7fdfaaffd640 1 -- 172.21.15.150:0/554532577 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0327f0 con 0x7fdfac0b4c20 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 -- 172.21.15.150:0/554532577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 msgr2=0x7fdfac0b9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 --2- 172.21.15.150:0/554532577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac0b9080 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0097a0 tx=0x7fdf9c02a180 comp rx=0 tx=0).stop 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 -- 172.21.15.150:0/554532577 shutdown_connections 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 --2- 172.21.15.150:0/554532577 >> v2:172.21.15.187:3300/0 conn(0x7fdfac0b46d0 0x7fdfac160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 --2- 172.21.15.150:0/554532577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac0b9080 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 -- 172.21.15.150:0/554532577 >> 172.21.15.150:0/554532577 conn(0x7fdfac0d6560 msgr2=0x7fdfac0d24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 -- 172.21.15.150:0/554532577 shutdown_connections 2024-04-25T14:02:51.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.698+0000 7fdfb179d640 1 -- 172.21.15.150:0/554532577 wait complete. 2024-04-25T14:02:51.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.700+0000 7fdfb179d640 1 Processor -- start 2024-04-25T14:02:51.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.700+0000 7fdfb179d640 1 -- start start 2024-04-25T14:02:51.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac15a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfac0d0970 0x7fdfac0b6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac0d15f0 con 0x7fdfac0b46d0 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfac06c8f0 con 0x7fdfac0d0970 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb179d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfac0d46b0 con 0x7fdfac0b4c20 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfabfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfabfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44720/0 (socket says 172.21.15.150:44720) 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfabfff640 1 -- 172.21.15.150:0/2100443327 learned_addr learned my addr 172.21.15.150:0/2100443327 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfab7fe640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac15a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.701+0000 7fdfb0f9c640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfac0d0970 0x7fdfac0b6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 -- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 msgr2=0x7fdfac15a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac15a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 -- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfac0d0970 msgr2=0x7fdfac0b6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfac0d0970 0x7fdfac0b6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 -- 172.21.15.150:0/2100443327 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfac14a180 con 0x7fdfac0b46d0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfabfff640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02a1c0 tx=0x7fdf9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c037040 con 0x7fdfac0b46d0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf9c042070 con 0x7fdfac0b46d0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfac1f64c0 con 0x7fdfac0b46d0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c03b930 con 0x7fdfac0b46d0 2024-04-25T14:02:51.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.702+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfac1fa260 con 0x7fdfac0b46d0 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.703+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfac050740 con 0x7fdfac0b46d0 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.704+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdf9c03bad0 con 0x7fdfac0b46d0 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.704+0000 7fdfa8ff9640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 0x7fdf980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.704+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fdf9c0b6980 con 0x7fdfac0b46d0 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.704+0000 7fdfab7fe640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 0x7fdf980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:51.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.705+0000 7fdfab7fe640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 0x7fdf980769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b73c0 tx=0x7fdfa0009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:51.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.708+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdf9c080750 con 0x7fdfac0b46d0 2024-04-25T14:02:51.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.885+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdfac0d3f80 con 0x7fdfac0b46d0 2024-04-25T14:02:51.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.886+0000 7fdfa8ff9640 1 -- 172.21.15.150:0/2100443327 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fdf9c084a50 con 0x7fdfac0b46d0 2024-04-25T14:02:51.885 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:51.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 msgr2=0x7fdf980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 0x7fdf980769c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdfac0b73c0 tx=0x7fdfa0009190 comp rx=0 tx=0).stop 2024-04-25T14:02:51.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 msgr2=0x7fdfac1f5f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:51.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c02a1c0 tx=0x7fdf9c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 shutdown_connections 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf980744b0 0x7fdf980769c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfac0d0970 0x7fdfac0b6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfac0b4c20 0x7fdfac15a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 --2- 172.21.15.150:0/2100443327 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfac0b46d0 0x7fdfac1f5f80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 >> 172.21.15.150:0/2100443327 conn(0x7fdfac0d6560 msgr2=0x7fdfac06d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 shutdown_connections 2024-04-25T14:02:51.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:51.888+0000 7fdfb179d640 1 -- 172.21.15.150:0/2100443327 wait complete. 2024-04-25T14:02:51.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:52.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.010+0000 7f86ec9b2640 1 Processor -- start 2024-04-25T14:02:52.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.010+0000 7f86ec9b2640 1 -- start start 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86ec9b2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e80b4c10 0x7f86e8160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86ec9b2640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f86e80b91f0 0x7f86e806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86ec9b2640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86e80b9600 con 0x7f86e806d630 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86ec9b2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f86e806cc60 con 0x7f86e80b4c10 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86ec9b2640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f86e80d4f20 con 0x7f86e80b91f0 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86e659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e80b4c10 0x7f86e8160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86e659c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86e80b4c10 0x7f86e8160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44730/0 (socket says 172.21.15.150:44730) 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.011+0000 7f86e659c640 1 -- 172.21.15.150:0/3007755252 learned_addr learned my addr 172.21.15.150:0/3007755252 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86dffff640 1 --2- 172.21.15.150:0/3007755252 >> v2:172.21.15.187:3300/0 conn(0x7f86e80b91f0 0x7f86e806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e659c640 1 -- 172.21.15.150:0/3007755252 >> v2:172.21.15.187:3300/0 conn(0x7f86e80b91f0 msgr2=0x7f86e806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e659c640 1 --2- 172.21.15.150:0/3007755252 >> v2:172.21.15.187:3300/0 conn(0x7f86e80b91f0 0x7f86e806d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e659c640 1 -- 172.21.15.150:0/3007755252 >> v1:172.21.15.187:6789/0 conn(0x7f86e806d630 legacy=0x7f86e8163c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e659c640 1 -- 172.21.15.150:0/3007755252 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e80d5200 con 0x7f86e80b4c10 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e659c640 1 --2- 172.21.15.150:0/3007755252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8160150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86e80c93d0 tx=0x7f86d802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a475f92c689e409a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.012+0000 7f86e5d9b640 1 -- 172.21.15.150:0/3007755252 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8037070 con 0x7f86e80b4c10 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86e5d9b640 1 -- 172.21.15.150:0/3007755252 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d8032440 con 0x7f86e80b4c10 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86e5d9b640 1 -- 172.21.15.150:0/3007755252 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d8032810 con 0x7f86e80b4c10 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 -- 172.21.15.150:0/3007755252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 msgr2=0x7f86e8160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/3007755252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8160150 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f86e80c93d0 tx=0x7f86d802a230 comp rx=0 tx=0).stop 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 -- 172.21.15.150:0/3007755252 shutdown_connections 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/3007755252 >> v2:172.21.15.187:3300/0 conn(0x7f86e80b91f0 0x7f86e806d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/3007755252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8160150 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 -- 172.21.15.150:0/3007755252 >> 172.21.15.150:0/3007755252 conn(0x7f86e80b7690 msgr2=0x7f86e80b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 -- 172.21.15.150:0/3007755252 shutdown_connections 2024-04-25T14:02:52.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.013+0000 7f86ec9b2640 1 -- 172.21.15.150:0/3007755252 wait complete. 2024-04-25T14:02:52.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.015+0000 7f86ec9b2640 1 Processor -- start 2024-04-25T14:02:52.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.015+0000 7f86ec9b2640 1 -- start start 2024-04-25T14:02:52.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86e806d630 0x7f86e80ac520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e80b91f0 0x7f86e81f9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86e806c8f0 con 0x7f86e80b4c10 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f86e806c610 con 0x7f86e80b91f0 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.016+0000 7f86ec9b2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f86e806da40 con 0x7f86e806d630 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44734/0 (socket says 172.21.15.150:44734) 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 -- 172.21.15.150:0/2585569221 learned_addr learned my addr 172.21.15.150:0/2585569221 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86e6d9d640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e80b91f0 0x7f86e81f9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86e659c640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86e806d630 0x7f86e80ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 -- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86e806d630 msgr2=0x7f86e80ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86e806d630 0x7f86e80ac520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 -- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e80b91f0 msgr2=0x7f86e81f9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e80b91f0 0x7f86e81f9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 -- 172.21.15.150:0/2585569221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86e8162730 con 0x7f86e80b4c10 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86dffff640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f86d000d5c0 tx=0x7f86d000da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00125f0 con 0x7f86e80b4c10 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d0019070 con 0x7f86e80b4c10 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86e8162a10 con 0x7f86e80b4c10 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00129d0 con 0x7f86e80b4c10 2024-04-25T14:02:52.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.017+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86e8162fd0 con 0x7f86e80b4c10 2024-04-25T14:02:52.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.018+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86e806ddd0 con 0x7f86e80b4c10 2024-04-25T14:02:52.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.019+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f86d0012b70 con 0x7f86e80b4c10 2024-04-25T14:02:52.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.020+0000 7f86deffd640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.020+0000 7f86e659c640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.020+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f86d0095870 con 0x7f86e80b4c10 2024-04-25T14:02:52.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.020+0000 7f86e659c640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 0x7f86c80769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f86d8034000 tx=0x7f86d802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.024+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86e806ddd0 con 0x7f86e80b4c10 2024-04-25T14:02:52.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.200+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f86e81624c0 con 0x7f86e80b4c10 2024-04-25T14:02:52.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.200+0000 7f86deffd640 1 -- 172.21.15.150:0/2585569221 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f86d005f5b0 con 0x7f86e80b4c10 2024-04-25T14:02:52.199 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:52.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 msgr2=0x7f86c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 0x7f86c80769c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f86d8034000 tx=0x7f86d802e000 comp rx=0 tx=0).stop 2024-04-25T14:02:52.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 msgr2=0x7f86e8147670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f86d000d5c0 tx=0x7f86d000da80 comp rx=0 tx=0).stop 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 shutdown_connections 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86c80744b0 0x7f86c80769c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86e80b91f0 0x7f86e81f9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86e80b4c10 0x7f86e8147670 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 --2- 172.21.15.150:0/2585569221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86e806d630 0x7f86e80ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 >> 172.21.15.150:0/2585569221 conn(0x7f86e80b7690 msgr2=0x7f86e806ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 shutdown_connections 2024-04-25T14:02:52.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.202+0000 7f86ec9b2640 1 -- 172.21.15.150:0/2585569221 wait complete. 2024-04-25T14:02:52.213 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.324+0000 7fccfbd8c640 1 Processor -- start 2024-04-25T14:02:52.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.324+0000 7fccfbd8c640 1 -- start start 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfbd8c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf40b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfbd8c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccf40ae7e0 0x7fccf40aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfbd8c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf40b6140 con 0x7fccf415df00 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfbd8c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fccf415e310 con 0x7fccf40ae7e0 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfbd8c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fccf40b5360 con 0x7fccf40b57f0 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfa329640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccf40ae7e0 0x7fccf40aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfa329640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fccf40ae7e0 0x7fccf40aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44740/0 (socket says 172.21.15.150:44740) 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.325+0000 7fccfa329640 1 -- 172.21.15.150:0/3198622003 learned_addr learned my addr 172.21.15.150:0/3198622003 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccf9327640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf40b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccfa329640 1 -- 172.21.15.150:0/3198622003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 msgr2=0x7fccf40b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccfa329640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf40b5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccfa329640 1 -- 172.21.15.150:0/3198622003 >> v1:172.21.15.187:6789/0 conn(0x7fccf415df00 legacy=0x7fccf406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccfa329640 1 -- 172.21.15.150:0/3198622003 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf40b4750 con 0x7fccf40ae7e0 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.326+0000 7fccfa329640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf40aebf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fccec00d150 tx=0x7fccec02c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d57eec23a9a548a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccf8b26640 1 -- 172.21.15.150:0/3198622003 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec039070 con 0x7fccf40ae7e0 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccf8b26640 1 -- 172.21.15.150:0/3198622003 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec009600 con 0x7fccf40ae7e0 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccf8b26640 1 -- 172.21.15.150:0/3198622003 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec0099d0 con 0x7fccf40ae7e0 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 -- 172.21.15.150:0/3198622003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 msgr2=0x7fccf40aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf40aebf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fccec00d150 tx=0x7fccec02c890 comp rx=0 tx=0).stop 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 -- 172.21.15.150:0/3198622003 shutdown_connections 2024-04-25T14:02:52.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf40aebf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/3198622003 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf40b5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 -- 172.21.15.150:0/3198622003 >> 172.21.15.150:0/3198622003 conn(0x7fccf40d1ae0 msgr2=0x7fccf40d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 -- 172.21.15.150:0/3198622003 shutdown_connections 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.327+0000 7fccfbd8c640 1 -- 172.21.15.150:0/3198622003 wait complete. 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.329+0000 7fccfbd8c640 1 Processor -- start 2024-04-25T14:02:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.329+0000 7fccfbd8c640 1 -- start start 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf41f21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccf415df00 0x7fccf415f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fccf40b4ff0 con 0x7fccf40ae7e0 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fccf40b4d10 con 0x7fccf415df00 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccfbd8c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fccf406f950 con 0x7fccf40b57f0 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.330+0000 7fccf9b28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44742/0 (socket says 172.21.15.150:44742) 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 -- 172.21.15.150:0/437334268 learned_addr learned my addr 172.21.15.150:0/437334268 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccfa329640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccf415df00 0x7fccf415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9327640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf41f21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 -- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 msgr2=0x7fccf41f21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf41f21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 -- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccf415df00 msgr2=0x7fccf415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccf415df00 0x7fccf415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 -- 172.21.15.150:0/437334268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf41f2c60 con 0x7fccf40ae7e0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccf9b28640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccf000c3c0 tx=0x7fccf000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf00052d0 con 0x7fccf40ae7e0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf0005490 con 0x7fccf40ae7e0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0007260 con 0x7fccf40ae7e0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf41f2f40 con 0x7fccf40ae7e0 2024-04-25T14:02:52.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.331+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf41f3500 con 0x7fccf40ae7e0 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.332+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf4076a40 con 0x7fccf40ae7e0 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.333+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fccf0007400 con 0x7fccf40ae7e0 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.333+0000 7fccea7fc640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 0x7fccd80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.333+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fccf00949c0 con 0x7fccf40ae7e0 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.333+0000 7fccf9327640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 0x7fccd80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.334+0000 7fccf9327640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 0x7fccd80769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fccf415a9a0 tx=0x7fcce4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.338+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fccf0096e20 con 0x7fccf40ae7e0 2024-04-25T14:02:52.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.513+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccf415aca0 con 0x7fccf40ae7e0 2024-04-25T14:02:52.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.515+0000 7fccea7fc640 1 -- 172.21.15.150:0/437334268 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fccf005e790 con 0x7fccf40ae7e0 2024-04-25T14:02:52.514 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:52.514 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.516+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 msgr2=0x7fccd80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 0x7fccd80769c0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fccf415a9a0 tx=0x7fcce4008040 comp rx=0 tx=0).stop 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 msgr2=0x7fccf414d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fccf000c3c0 tx=0x7fccf000c880 comp rx=0 tx=0).stop 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 shutdown_connections 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fccd80744b0 0x7fccd80769c0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fccf415df00 0x7fccf415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fccf40b57f0 0x7fccf41f21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 --2- 172.21.15.150:0/437334268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fccf40ae7e0 0x7fccf414d3a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 >> 172.21.15.150:0/437334268 conn(0x7fccf40d1ae0 msgr2=0x7fccf415d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 shutdown_connections 2024-04-25T14:02:52.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.517+0000 7fccfbd8c640 1 -- 172.21.15.150:0/437334268 wait complete. 2024-04-25T14:02:52.527 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:52.528 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:52.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.639+0000 7fe847e67640 1 Processor -- start 2024-04-25T14:02:52.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.640+0000 7fe847e67640 1 -- start start 2024-04-25T14:02:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe847e67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe840070820 0x7fe8401609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe847e67640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe840160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe847e67640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8400cf300 con 0x7fe8400d03a0 2024-04-25T14:02:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe847e67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe8400cff10 con 0x7fe840070820 2024-04-25T14:02:52.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe847e67640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8400cf5e0 con 0x7fe8400cdca0 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe845c03640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe8400d03a0 0x7fe8400cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55184/0 (socket says 172.21.15.150:55184) 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe845c03640 1 -- 172.21.15.150:0/3847723712 learned_addr learned my addr 172.21.15.150:0/3847723712 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe845402640 1 --2- 172.21.15.150:0/3847723712 >> v2:172.21.15.150:3300/0 conn(0x7fe840070820 0x7fe8401609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.641+0000 7fe846404640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe840160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe845402640 1 -- 172.21.15.150:0/3847723712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 msgr2=0x7fe840160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe845402640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe840160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe845402640 1 -- 172.21.15.150:0/3847723712 >> v1:172.21.15.150:6789/0 conn(0x7fe8400d03a0 legacy=0x7fe8400cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe845402640 1 -- 172.21.15.150:0/3847723712 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8400cf8c0 con 0x7fe840070820 2024-04-25T14:02:52.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe845402640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe840070820 0x7fe8401609e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe8300097a0 tx=0x7fe83002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9986bcc0550d2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.642+0000 7fe844c01640 1 -- 172.21.15.150:0/3847723712 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830037070 con 0x7fe840070820 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe844c01640 1 -- 172.21.15.150:0/3847723712 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe830032440 con 0x7fe840070820 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe844c01640 1 -- 172.21.15.150:0/3847723712 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe830032810 con 0x7fe840070820 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe847e67640 1 -- 172.21.15.150:0/3847723712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe840070820 msgr2=0x7fe8401609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe847e67640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe840070820 0x7fe8401609e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe8300097a0 tx=0x7fe83002a180 comp rx=0 tx=0).stop 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe847e67640 1 -- 172.21.15.150:0/3847723712 shutdown_connections 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe847e67640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe840160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.643+0000 7fe847e67640 1 --2- 172.21.15.150:0/3847723712 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe840070820 0x7fe8401609e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.644+0000 7fe847e67640 1 -- 172.21.15.150:0/3847723712 >> 172.21.15.150:0/3847723712 conn(0x7fe8400ce390 msgr2=0x7fe8400ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.644+0000 7fe847e67640 1 -- 172.21.15.150:0/3847723712 shutdown_connections 2024-04-25T14:02:52.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.644+0000 7fe847e67640 1 -- 172.21.15.150:0/3847723712 wait complete. 2024-04-25T14:02:52.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.645+0000 7fe847e67640 1 Processor -- start 2024-04-25T14:02:52.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.645+0000 7fe847e67640 1 -- start start 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe840070820 0x7fe840144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe8401f4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe84006f190 con 0x7fe8400d03a0 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe84006f010 con 0x7fe840070820 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe84006f310 con 0x7fe8400cdca0 2024-04-25T14:02:52.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44770/0 (socket says 172.21.15.150:44770) 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 -- 172.21.15.150:0/2954998632 learned_addr learned my addr 172.21.15.150:0/2954998632 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe845402640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe8401f4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe845c03640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe840070820 0x7fe840144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 -- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 msgr2=0x7fe8401f4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe8401f4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 -- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe840070820 msgr2=0x7fe840144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe840070820 0x7fe840144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 -- 172.21.15.150:0/2954998632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8401f5270 con 0x7fe8400d03a0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe846404640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe83800d5d0 tx=0x7fe83800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8380125f0 con 0x7fe8400d03a0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe838019070 con 0x7fe8400d03a0 2024-04-25T14:02:52.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8401477e0 con 0x7fe8400d03a0 2024-04-25T14:02:52.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.647+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8380129f0 con 0x7fe8400d03a0 2024-04-25T14:02:52.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.648+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe840147da0 con 0x7fe8400d03a0 2024-04-25T14:02:52.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.649+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe840147530 con 0x7fe8400d03a0 2024-04-25T14:02:52.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.649+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe838012b90 con 0x7fe8400d03a0 2024-04-25T14:02:52.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.650+0000 7fe8367fc640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 0x7fe8240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.650+0000 7fe845c03640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 0x7fe8240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.650+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe8380959a0 con 0x7fe8400d03a0 2024-04-25T14:02:52.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.650+0000 7fe845c03640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 0x7fe8240769c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0012e0 tx=0x7fe83c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.655+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe83805fdd0 con 0x7fe8400d03a0 2024-04-25T14:02:52.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.831+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe840076a40 con 0x7fe8400d03a0 2024-04-25T14:02:52.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.832+0000 7fe8367fc640 1 -- 172.21.15.150:0/2954998632 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fe83805f6d0 con 0x7fe8400d03a0 2024-04-25T14:02:52.831 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 msgr2=0x7fe8240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 0x7fe8240769c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0012e0 tx=0x7fe83c009040 comp rx=0 tx=0).stop 2024-04-25T14:02:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 msgr2=0x7fe8400b00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe83800d5d0 tx=0x7fe83800da90 comp rx=0 tx=0).stop 2024-04-25T14:02:52.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 shutdown_connections 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8240744b0 0x7fe8240769c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8400d03a0 0x7fe8400b00e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8400cdca0 0x7fe8401f4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 --2- 172.21.15.150:0/2954998632 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe840070820 0x7fe840144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 >> 172.21.15.150:0/2954998632 conn(0x7fe8400ce390 msgr2=0x7fe840161110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 shutdown_connections 2024-04-25T14:02:52.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.834+0000 7fe847e67640 1 -- 172.21.15.150:0/2954998632 wait complete. 2024-04-25T14:02:52.845 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:52.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.956+0000 7f776d814640 1 Processor -- start 2024-04-25T14:02:52.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.956+0000 7f776d814640 1 -- start start 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.957+0000 7f776d814640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77680b4c10 0x7f7768160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f776d814640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f776806d800 0x7f77680d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f776d814640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77680b9600 con 0x7f77680b91f0 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f776d814640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f776806cc60 con 0x7f776806d800 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f776d814640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f77680d4f20 con 0x7f77680b4c10 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f776806d800 0x7f77680d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f776806d800 0x7f77680d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44778/0 (socket says 172.21.15.150:44778) 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 -- 172.21.15.150:0/1505358657 learned_addr learned my addr 172.21.15.150:0/1505358657 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f7766ffd640 1 --2- 172.21.15.150:0/1505358657 >> v2:172.21.15.187:3300/0 conn(0x7f77680b4c10 0x7f7768160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 -- 172.21.15.150:0/1505358657 >> v2:172.21.15.187:3300/0 conn(0x7f77680b4c10 msgr2=0x7f7768160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 --2- 172.21.15.150:0/1505358657 >> v2:172.21.15.187:3300/0 conn(0x7f77680b4c10 0x7f7768160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 -- 172.21.15.150:0/1505358657 >> v1:172.21.15.187:6789/0 conn(0x7f77680b91f0 legacy=0x7f776806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.958+0000 7f77677fe640 1 -- 172.21.15.150:0/1505358657 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77680d5200 con 0x7f776806d800 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.959+0000 7f77677fe640 1 --2- 172.21.15.150:0/1505358657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f776806d800 0x7f77680d0950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f776808e810 tx=0x7f776002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31586cf483515217 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.959+0000 7f7765ffb640 1 -- 172.21.15.150:0/1505358657 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760036070 con 0x7f776806d800 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.959+0000 7f7765ffb640 1 -- 172.21.15.150:0/1505358657 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7760031600 con 0x7f776806d800 2024-04-25T14:02:52.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.959+0000 7f7765ffb640 1 -- 172.21.15.150:0/1505358657 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77600319d0 con 0x7f776806d800 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 -- 172.21.15.150:0/1505358657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f776806d800 msgr2=0x7f77680d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 --2- 172.21.15.150:0/1505358657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f776806d800 0x7f77680d0950 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f776808e810 tx=0x7f776002a6f0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 -- 172.21.15.150:0/1505358657 shutdown_connections 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 --2- 172.21.15.150:0/1505358657 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f776806d800 0x7f77680d0950 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 --2- 172.21.15.150:0/1505358657 >> v2:172.21.15.187:3300/0 conn(0x7f77680b4c10 0x7f7768160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 -- 172.21.15.150:0/1505358657 >> 172.21.15.150:0/1505358657 conn(0x7f77680b7690 msgr2=0x7f77680b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 -- 172.21.15.150:0/1505358657 shutdown_connections 2024-04-25T14:02:52.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.960+0000 7f776d814640 1 -- 172.21.15.150:0/1505358657 wait complete. 2024-04-25T14:02:52.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.961+0000 7f776d814640 1 Processor -- start 2024-04-25T14:02:52.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.962+0000 7f776d814640 1 -- start start 2024-04-25T14:02:52.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776806d800 0x7f776814b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b91f0 0x7f7768161f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f776806c8f0 con 0x7f77680b4c10 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f776806c610 con 0x7f776806d800 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f776d814640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f776806dc10 con 0x7f77680b91f0 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44794/0 (socket says 172.21.15.150:44794) 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 -- 172.21.15.150:0/2713433423 learned_addr learned my addr 172.21.15.150:0/2713433423 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77677fe640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b91f0 0x7f7768161f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f7766ffd640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776806d800 0x7f776814b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 -- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b91f0 msgr2=0x7f7768161f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b91f0 0x7f7768161f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 -- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776806d800 msgr2=0x7f776814b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776806d800 0x7f776814b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.963+0000 7f77667fc640 1 -- 172.21.15.150:0/2713433423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7768149d80 con 0x7f77680b4c10 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f77667fc640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f775c00b3c0 tx=0x7f775c00b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c011830 con 0x7f77680b4c10 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775c016070 con 0x7f77680b4c10 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776814a060 con 0x7f77680b4c10 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c011c10 con 0x7f77680b4c10 2024-04-25T14:02:52.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.964+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f776814a620 con 0x7f77680b4c10 2024-04-25T14:02:52.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.965+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7768076a40 con 0x7f77680b4c10 2024-04-25T14:02:52.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.965+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f775c01b460 con 0x7f77680b4c10 2024-04-25T14:02:52.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.966+0000 7f774ffff640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:52.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.966+0000 7f7766ffd640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:52.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.966+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f775c095030 con 0x7f77680b4c10 2024-04-25T14:02:52.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.966+0000 7f7766ffd640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7758006fd0 tx=0x7f7758008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:52.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:52.971+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f775c020020 con 0x7f77680b4c10 2024-04-25T14:02:53.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.145+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7768149ad0 con 0x7f77680b4c10 2024-04-25T14:02:53.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.146+0000 7f774ffff640 1 -- 172.21.15.150:0/2713433423 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f775c05f450 con 0x7f77680b4c10 2024-04-25T14:02:53.145 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.147+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 msgr2=0x7f77540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.147+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7758006fd0 tx=0x7f7758008040 comp rx=0 tx=0).stop 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 msgr2=0x7f7768149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f775c00b3c0 tx=0x7f775c00b880 comp rx=0 tx=0).stop 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 shutdown_connections 2024-04-25T14:02:53.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b91f0 0x7f7768161f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b4c10 0x7f7768149050 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 --2- 172.21.15.150:0/2713433423 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f776806d800 0x7f776814b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 >> 172.21.15.150:0/2713433423 conn(0x7f77680b7690 msgr2=0x7f7768163020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 shutdown_connections 2024-04-25T14:02:53.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.148+0000 7f776d814640 1 -- 172.21.15.150:0/2713433423 wait complete. 2024-04-25T14:02:53.158 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:53.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.272+0000 7f8b77572640 1 Processor -- start 2024-04-25T14:02:53.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.272+0000 7f8b77572640 1 -- start start 2024-04-25T14:02:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b77572640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8b700cdfe0 0x7f8b700b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b77572640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b77572640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b700d3570 con 0x7f8b700cf8e0 2024-04-25T14:02:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b77572640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8b700b4800 con 0x7f8b700cdfe0 2024-04-25T14:02:53.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b77572640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b700b4ae0 con 0x7f8b701605d0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b7530e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8b700cf8e0 0x7f8b700b3a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55188/0 (socket says 172.21.15.150:55188) 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b7530e640 1 -- 172.21.15.150:0/3122335721 learned_addr learned my addr 172.21.15.150:0/3122335721 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/3122335721 >> v2:172.21.15.187:3300/0 conn(0x7f8b700cdfe0 0x7f8b700b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b75b0f640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3123461398 0 0) 0x7f8b700d3570 con 0x7f8b700cf8e0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b5c003520 con 0x7f8b700cf8e0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1499678389 0 0) 0x7f8b5c003520 con 0x7f8b700cf8e0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b74b0d640 1 -- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 msgr2=0x7f8b700d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b74b0d640 1 -- 172.21.15.150:0/3122335721 >> v1:172.21.15.150:6789/0 conn(0x7f8b700cf8e0 legacy=0x7f8b700b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b74b0d640 1 -- 172.21.15.150:0/3122335721 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b700b4dc0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.274+0000 7f8b75b0f640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.275+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cdfe0 0x7f8b700b4110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b600097a0 tx=0x7f8b6002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=361b93fcc0b3f8ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.275+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b60037070 con 0x7f8b700cdfe0 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.275+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6002aba0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.275+0000 7f8b67fff640 1 -- 172.21.15.150:0/3122335721 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c0019f0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 -- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cdfe0 msgr2=0x7f8b700b4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cdfe0 0x7f8b700b4110 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8b600097a0 tx=0x7f8b6002a110 comp rx=0 tx=0).stop 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 -- 172.21.15.150:0/3122335721 shutdown_connections 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 --2- 172.21.15.150:0/3122335721 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cdfe0 0x7f8b700b4110 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 -- 172.21.15.150:0/3122335721 >> 172.21.15.150:0/3122335721 conn(0x7f8b7009b4a0 msgr2=0x7f8b7009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 -- 172.21.15.150:0/3122335721 shutdown_connections 2024-04-25T14:02:53.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.276+0000 7f8b77572640 1 -- 172.21.15.150:0/3122335721 wait complete. 2024-04-25T14:02:53.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.278+0000 7f8b77572640 1 Processor -- start 2024-04-25T14:02:53.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.278+0000 7f8b77572640 1 -- start start 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 0x7f8b70070ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b700d2360 con 0x7f8b700cdfe0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b700d21e0 con 0x7f8b700cf8e0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b77572640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b700d24e0 con 0x7f8b701605d0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44802/0 (socket says 172.21.15.150:44802) 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 -- 172.21.15.150:0/1956815358 learned_addr learned my addr 172.21.15.150:0/1956815358 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 0x7f8b70070ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b75b0f640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 -- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 msgr2=0x7f8b700c7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700c7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 -- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 msgr2=0x7f8b70070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 0x7f8b70070ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b7530e640 1 -- 172.21.15.150:0/1956815358 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7014c1d0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b75b0f640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.279+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 0x7f8b70070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b7530e640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00cc70 tx=0x7f8b6c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c017070 con 0x7f8b700cdfe0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b6c012400 con 0x7f8b700cdfe0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b6c0127b0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b701faa80 con 0x7f8b700cdfe0 2024-04-25T14:02:53.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.280+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b701fa1e0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.281+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b70161520 con 0x7f8b700cdfe0 2024-04-25T14:02:53.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.282+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8b6c01a070 con 0x7f8b700cdfe0 2024-04-25T14:02:53.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.282+0000 7f8b65ffb640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 0x7f8b540769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.282+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8b6c095350 con 0x7f8b700cdfe0 2024-04-25T14:02:53.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.282+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 0x7f8b540769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.283+0000 7f8b74b0d640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 0x7f8b540769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b60034000 tx=0x7f8b6002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.286 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.287+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b70161520 con 0x7f8b700cdfe0 2024-04-25T14:02:53.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.471+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8b70076a40 con 0x7f8b700cdfe0 2024-04-25T14:02:53.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.472+0000 7f8b65ffb640 1 -- 172.21.15.150:0/1956815358 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f8b6c05f1b0 con 0x7f8b700cdfe0 2024-04-25T14:02:53.471 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 msgr2=0x7f8b540769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 0x7f8b540769c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f8b60034000 tx=0x7f8b6002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 msgr2=0x7f8b7014bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8b6c00cc70 tx=0x7f8b6c00a430 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 shutdown_connections 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b540744b0 0x7f8b540769c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b701605d0 0x7f8b700c7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b700cf8e0 0x7f8b70070ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 --2- 172.21.15.150:0/1956815358 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b700cdfe0 0x7f8b7014bc90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 >> 172.21.15.150:0/1956815358 conn(0x7f8b7009b4a0 msgr2=0x7f8b700684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 shutdown_connections 2024-04-25T14:02:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.474+0000 7f8b77572640 1 -- 172.21.15.150:0/1956815358 wait complete. 2024-04-25T14:02:53.484 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:53.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.596+0000 7f00bacb6640 1 Processor -- start 2024-04-25T14:02:53.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.596+0000 7f00bacb6640 1 -- start start 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00bacb6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00b4160460 0x7f00b406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00bacb6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00b4163d80 0x7f00b40b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00bacb6640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00b406ff50 con 0x7f00b40b5900 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00bacb6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f00b40b5d10 con 0x7f00b4163d80 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00bacb6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00b406f3d0 con 0x7f00b4160460 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00b3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00b4163d80 0x7f00b40b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00b3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f00b4163d80 0x7f00b40b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44804/0 (socket says 172.21.15.150:44804) 2024-04-25T14:02:53.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.597+0000 7f00b3fff640 1 -- 172.21.15.150:0/3094815866 learned_addr learned my addr 172.21.15.150:0/3094815866 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b8a52640 1 --2- 172.21.15.150:0/3094815866 >> v2:172.21.15.187:3300/0 conn(0x7f00b4160460 0x7f00b406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b3fff640 1 -- 172.21.15.150:0/3094815866 >> v2:172.21.15.187:3300/0 conn(0x7f00b4160460 msgr2=0x7f00b406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b3fff640 1 --2- 172.21.15.150:0/3094815866 >> v2:172.21.15.187:3300/0 conn(0x7f00b4160460 0x7f00b406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b3fff640 1 -- 172.21.15.150:0/3094815866 >> v1:172.21.15.187:6789/0 conn(0x7f00b40b5900 legacy=0x7f00b4072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b3fff640 1 -- 172.21.15.150:0/3094815866 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b406f6b0 con 0x7f00b4163d80 2024-04-25T14:02:53.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.598+0000 7f00b3fff640 1 --2- 172.21.15.150:0/3094815866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b4163d80 0x7f00b40b53c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f00b40d4950 tx=0x7f00a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c9f646736b0d01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00b37fe640 1 -- 172.21.15.150:0/3094815866 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0037070 con 0x7f00b4163d80 2024-04-25T14:02:53.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00b37fe640 1 -- 172.21.15.150:0/3094815866 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a0032440 con 0x7f00b4163d80 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00b37fe640 1 -- 172.21.15.150:0/3094815866 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a0032810 con 0x7f00b4163d80 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 -- 172.21.15.150:0/3094815866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b4163d80 msgr2=0x7f00b40b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 --2- 172.21.15.150:0/3094815866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b4163d80 0x7f00b40b53c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f00b40d4950 tx=0x7f00a002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 -- 172.21.15.150:0/3094815866 shutdown_connections 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 --2- 172.21.15.150:0/3094815866 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b4163d80 0x7f00b40b53c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 --2- 172.21.15.150:0/3094815866 >> v2:172.21.15.187:3300/0 conn(0x7f00b4160460 0x7f00b406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 -- 172.21.15.150:0/3094815866 >> 172.21.15.150:0/3094815866 conn(0x7f00b40d03c0 msgr2=0x7f00b40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 -- 172.21.15.150:0/3094815866 shutdown_connections 2024-04-25T14:02:53.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.599+0000 7f00bacb6640 1 -- 172.21.15.150:0/3094815866 wait complete. 2024-04-25T14:02:53.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.601+0000 7f00bacb6640 1 Processor -- start 2024-04-25T14:02:53.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.601+0000 7f00bacb6640 1 -- start start 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00b4163d80 0x7f00b4157b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00b415c900 0x7f00b4162630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00b40b3030 con 0x7f00b40b5900 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00b406fc70 con 0x7f00b4163d80 2024-04-25T14:02:53.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00b40ce5c0 con 0x7f00b415c900 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44816/0 (socket says 172.21.15.150:44816) 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 -- 172.21.15.150:0/2162897346 learned_addr learned my addr 172.21.15.150:0/2162897346 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b3fff640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00b4163d80 0x7f00b4157b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b9253640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00b415c900 0x7f00b4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 -- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00b415c900 msgr2=0x7f00b4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00b415c900 0x7f00b4162630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 -- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00b4163d80 msgr2=0x7f00b4157b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00b4163d80 0x7f00b4157b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 -- 172.21.15.150:0/2162897346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00b41f6150 con 0x7f00b40b5900 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b8a52640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00a800e8a0 tx=0x7f00a800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a8017070 con 0x7f00b40b5900 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00a800ee90 con 0x7f00b40b5900 2024-04-25T14:02:53.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00b40b7850 con 0x7f00b40b5900 2024-04-25T14:02:53.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00a800cb40 con 0x7f00b40b5900 2024-04-25T14:02:53.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.603+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00b40b7e10 con 0x7f00b40b5900 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.605+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00b4076a40 con 0x7f00b40b5900 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.605+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f00a801a070 con 0x7f00b40b5900 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.606+0000 7f00b17fa640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 0x7f00980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.606+0000 7f00b3fff640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 0x7f00980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.606+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f00a80953f0 con 0x7f00b40b5900 2024-04-25T14:02:53.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.606+0000 7f00b3fff640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 0x7f00980769c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00a0034000 tx=0x7f00a002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.610+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00a805f1c0 con 0x7f00b40b5900 2024-04-25T14:02:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.788+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f00b40b98c0 con 0x7f00b40b5900 2024-04-25T14:02:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.789+0000 7f00b17fa640 1 -- 172.21.15.150:0/2162897346 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f00a80634c0 con 0x7f00b40b5900 2024-04-25T14:02:53.788 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 msgr2=0x7f00980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 0x7f00980769c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00a0034000 tx=0x7f00a002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 msgr2=0x7f00b415ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f00a800e8a0 tx=0x7f00a800ed60 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 shutdown_connections 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00980744b0 0x7f00980769c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00b415c900 0x7f00b4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00b4163d80 0x7f00b4157b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 --2- 172.21.15.150:0/2162897346 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00b40b5900 0x7f00b415ee50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 >> 172.21.15.150:0/2162897346 conn(0x7f00b40d03c0 msgr2=0x7f00b4072f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 shutdown_connections 2024-04-25T14:02:53.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.791+0000 7f00bacb6640 1 -- 172.21.15.150:0/2162897346 wait complete. 2024-04-25T14:02:53.802 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:53.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.911+0000 7fee1592d640 1 Processor -- start 2024-04-25T14:02:53.910 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.911+0000 7fee1592d640 1 -- start start 2024-04-25T14:02:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee1592d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee101601d0 0x7fee100d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee1592d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee1006c570 con 0x7fee1006da00 2024-04-25T14:02:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee1592d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee101605e0 con 0x7fee100b9070 2024-04-25T14:02:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee1592d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fee100d5980 con 0x7fee101601d0 2024-04-25T14:02:53.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee0f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fee1006da00 0x7fee10163ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55198/0 (socket says 172.21.15.150:55198) 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.912+0000 7fee0f7fe640 1 -- 172.21.15.150:0/2723612794 learned_addr learned my addr 172.21.15.150:0/2723612794 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0effd640 1 --2- 172.21.15.150:0/2723612794 >> v2:172.21.15.187:3300/0 conn(0x7fee101601d0 0x7fee100d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972019856 0 0) 0x7fee1006c570 con 0x7fee1006da00 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fedf8003520 con 0x7fee1006da00 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3913308038 0 0) 0x7fee101605e0 con 0x7fee100b9070 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee1006c570 con 0x7fee100b9070 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0effd640 1 -- 172.21.15.150:0/2723612794 >> v1:172.21.15.187:6789/0 conn(0x7fee100b9070 legacy=0x7fee1006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0effd640 1 -- 172.21.15.150:0/2723612794 >> v1:172.21.15.150:6789/0 conn(0x7fee1006da00 legacy=0x7fee10163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.913+0000 7fee0effd640 1 -- 172.21.15.150:0/2723612794 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1006c810 con 0x7fee101601d0 2024-04-25T14:02:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.914+0000 7fee0effd640 1 --2- 172.21.15.150:0/2723612794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee101601d0 0x7fee100d5100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fee000097a0 tx=0x7fee0002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1254b8a326f91c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.914+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee00037070 con 0x7fee101601d0 2024-04-25T14:02:53.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.914+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee00032440 con 0x7fee101601d0 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.914+0000 7fee0dffb640 1 -- 172.21.15.150:0/2723612794 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee000327f0 con 0x7fee101601d0 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 -- 172.21.15.150:0/2723612794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee101601d0 msgr2=0x7fee100d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 --2- 172.21.15.150:0/2723612794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee101601d0 0x7fee100d5100 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fee000097a0 tx=0x7fee0002a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 -- 172.21.15.150:0/2723612794 shutdown_connections 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 --2- 172.21.15.150:0/2723612794 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee101601d0 0x7fee100d5100 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 -- 172.21.15.150:0/2723612794 >> 172.21.15.150:0/2723612794 conn(0x7fee100d5d90 msgr2=0x7fee100d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 -- 172.21.15.150:0/2723612794 shutdown_connections 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.915+0000 7fee1592d640 1 -- 172.21.15.150:0/2723612794 wait complete. 2024-04-25T14:02:53.915 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.916+0000 7fee1592d640 1 Processor -- start 2024-04-25T14:02:53.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.917+0000 7fee1592d640 1 -- start start 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee1006da00 0x7fee10152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 0x7fee100ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee100d5640 con 0x7fee100b9070 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee1006cdd0 con 0x7fee1006da00 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee1592d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee1006de10 con 0x7fee101601d0 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44822/0 (socket says 172.21.15.150:44822) 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 -- 172.21.15.150:0/2504846737 learned_addr learned my addr 172.21.15.150:0/2504846737 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:53.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0effd640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee1006da00 0x7fee10152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0f7fe640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 0x7fee100ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 -- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 msgr2=0x7fee100ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 0x7fee100ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 -- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee1006da00 msgr2=0x7fee10152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee1006da00 0x7fee10152510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0e7fc640 1 -- 172.21.15.150:0/2504846737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee10162720 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.918+0000 7fee0f7fe640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 0x7fee100ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fee0e7fc640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c3c0 tx=0x7fee0400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04017070 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee040052f0 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee10162a00 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04003830 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.919+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee10162fc0 con 0x7fee100b9070 2024-04-25T14:02:53.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.920+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee10076a40 con 0x7fee100b9070 2024-04-25T14:02:53.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.921+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee0401a070 con 0x7fee100b9070 2024-04-25T14:02:53.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.921+0000 7fedf7fff640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 0x7fedfc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:53.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.921+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fee040949b0 con 0x7fee100b9070 2024-04-25T14:02:53.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.921+0000 7fee0effd640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 0x7fedfc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:53.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.922+0000 7fee0effd640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 0x7fedfc0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee00034000 tx=0x7fee0002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:53.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:53.926+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee0405e700 con 0x7fee100b9070 2024-04-25T14:02:54.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.102+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee100c9660 con 0x7fee100b9070 2024-04-25T14:02:54.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.103+0000 7fedf7fff640 1 -- 172.21.15.150:0/2504846737 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fee04062a00 con 0x7fee100b9070 2024-04-25T14:02:54.103 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:54.103 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:54.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 msgr2=0x7fedfc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 0x7fedfc0769c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fee00034000 tx=0x7fee0002e000 comp rx=0 tx=0).stop 2024-04-25T14:02:54.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 msgr2=0x7fee100ca3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee0400c3c0 tx=0x7fee0400c880 comp rx=0 tx=0).stop 2024-04-25T14:02:54.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 shutdown_connections 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fedfc0744b0 0x7fedfc0769c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee101601d0 0x7fee100ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee100b9070 0x7fee100ca3e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 --2- 172.21.15.150:0/2504846737 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee1006da00 0x7fee10152510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 >> 172.21.15.150:0/2504846737 conn(0x7fee100d5d90 msgr2=0x7fee100d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 shutdown_connections 2024-04-25T14:02:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.105+0000 7fee1592d640 1 -- 172.21.15.150:0/2504846737 wait complete. 2024-04-25T14:02:54.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:54.117 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:54.230 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.231+0000 7fa313b6c640 1 Processor -- start 2024-04-25T14:02:54.231 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.231+0000 7fa313b6c640 1 -- start start 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa313b6c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c0b2f20 0x7fa30c162300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa313b6c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30c15e7f0 con 0x7fa30c0735b0 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa313b6c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa30c162920 con 0x7fa30c15f2b0 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa313b6c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa30c15ee20 con 0x7fa30c0b2f20 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa311107640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa30c0735b0 0x7fa30c161bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55206/0 (socket says 172.21.15.150:55206) 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.232+0000 7fa311107640 1 -- 172.21.15.150:0/57976562 learned_addr learned my addr 172.21.15.150:0/57976562 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa312109640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c0b2f20 0x7fa30c162300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.235 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3800518600 0 0) 0x7fa30c15e7f0 con 0x7fa30c0735b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa2f8003520 con 0x7fa30c0735b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163320973 0 0) 0x7fa30c162920 con 0x7fa30c15f2b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa30c15e7f0 con 0x7fa30c15f2b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 355389263 0 0) 0x7fa2f8003520 con 0x7fa30c0735b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c0b2f20 msgr2=0x7fa30c162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c0b2f20 0x7fa30c162300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 >> v1:172.21.15.187:6789/0 conn(0x7fa30c15f2b0 legacy=0x7fa30c072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c15eab0 con 0x7fa30c0735b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fa2fc001670 con 0x7fa30c0735b0 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 >> v1:172.21.15.150:6789/0 conn(0x7fa30c0735b0 legacy=0x7fa30c161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c15ee20 con 0x7fa2f8003b70 2024-04-25T14:02:54.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f800c140 con 0x7fa2f80051c0 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f800c340 con 0x7fa2f8004800 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa312109640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c0b2f20 0x7fa30c162300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311908640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311107640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa312109640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311908640 1 -- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f8004800 msgr2=0x7fa2f8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311908640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311908640 1 -- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80051c0 msgr2=0x7fa2f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.233+0000 7fa311908640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80051c0 0x7fa2f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa311908640 1 -- 172.21.15.150:0/57976562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f8003520 con 0x7fa2f8003b70 2024-04-25T14:02:54.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa311107640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f8004800 0x7fa2f8004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa311908640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa308001890 tx=0x7fa2f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4a32f199849e9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3080014e0 con 0x7fa2f8003b70 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3080052f0 con 0x7fa2f8003b70 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.234+0000 7fa310906640 1 -- 172.21.15.150:0/57976562 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30802ac60 con 0x7fa2f8003b70 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 -- 172.21.15.150:0/57976562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 msgr2=0x7fa2f8003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 --2- 172.21.15.150:0/57976562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa2f8003fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa308001890 tx=0x7fa2f8005830 comp rx=0 tx=0).stop 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa312109640 1 -- 172.21.15.150:0/57976562 reap_dead start 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 -- 172.21.15.150:0/57976562 shutdown_connections 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 -- 172.21.15.150:0/57976562 >> 172.21.15.150:0/57976562 conn(0x7fa30c0d39b0 msgr2=0x7fa30c0d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 -- 172.21.15.150:0/57976562 shutdown_connections 2024-04-25T14:02:54.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.235+0000 7fa313b6c640 1 -- 172.21.15.150:0/57976562 wait complete. 2024-04-25T14:02:54.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.237+0000 7fa313b6c640 1 Processor -- start 2024-04-25T14:02:54.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.237+0000 7fa313b6c640 1 -- start start 2024-04-25T14:02:54.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa30c153430 0x7fa30c073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c153840 0x7fa30c0737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c15d600 con 0x7fa2f8003b70 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa30c162550 con 0x7fa30c153430 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa313b6c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa30c15d780 con 0x7fa30c153840 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa311908640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa311908640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44836/0 (socket says 172.21.15.150:44836) 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa311908640 1 -- 172.21.15.150:0/4244480420 learned_addr learned my addr 172.21.15.150:0/4244480420 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa311908640 1 -- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c153840 msgr2=0x7fa30c0737d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa311107640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa30c153430 0x7fa30c073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.238+0000 7fa312109640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c153840 0x7fa30c0737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa311908640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c153840 0x7fa30c0737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa311908640 1 -- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa30c153430 msgr2=0x7fa30c073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa311908640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa30c153430 0x7fa30c073290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa311908640 1 -- 172.21.15.150:0/4244480420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa30c152370 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa311908640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa308008fd0 tx=0x7fa3080012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa308007a50 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa308041070 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3080054a0 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa30c154bd0 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.239+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa30c155190 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.240+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa30c1553d0 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.241+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa3080067d0 con 0x7fa2f8003b70 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.241+0000 7fa3027fc640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 0x7fa2f0076b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.241+0000 7fa311107640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 0x7fa2f0076b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.241+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa3080b5a60 con 0x7fa2f8003b70 2024-04-25T14:02:54.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.242+0000 7fa311107640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 0x7fa2f0076b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc000c00 tx=0x7fa2fc000ed0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.246+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa30c1553d0 con 0x7fa2f8003b70 2024-04-25T14:02:54.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.421+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa30c0ca7a0 con 0x7fa2f8003b70 2024-04-25T14:02:54.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.421+0000 7fa3027fc640 1 -- 172.21.15.150:0/4244480420 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fa30c0ca7a0 con 0x7fa2f8003b70 2024-04-25T14:02:54.420 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:54.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 msgr2=0x7fa2f0076b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 0x7fa2f0076b60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc000c00 tx=0x7fa2fc000ed0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 msgr2=0x7fa30c151e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa308008fd0 tx=0x7fa3080012d0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 shutdown_connections 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2f0074650 0x7fa2f0076b60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa30c153840 0x7fa30c0737d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa30c153430 0x7fa30c073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 --2- 172.21.15.150:0/4244480420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f8003b70 0x7fa30c151e30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 >> 172.21.15.150:0/4244480420 conn(0x7fa30c0d39b0 msgr2=0x7fa30c161bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 shutdown_connections 2024-04-25T14:02:54.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.423+0000 7fa313b6c640 1 -- 172.21.15.150:0/4244480420 wait complete. 2024-04-25T14:02:54.433 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:54.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.545+0000 7f19e229f640 1 Processor -- start 2024-04-25T14:02:54.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.545+0000 7f19e229f640 1 -- start start 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19e229f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19dc1600e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19e229f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19dc0d54f0 con 0x7f19dc0d1510 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19e229f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19dc06d1d0 con 0x7f19dc0d1080 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19e229f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19dc06cf80 con 0x7f19dc0ba190 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19dbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f19dc0d1510 0x7f19dc0d0970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55220/0 (socket says 172.21.15.150:55220) 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19dbfff640 1 -- 172.21.15.150:0/4170961254 learned_addr learned my addr 172.21.15.150:0/4170961254 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19e083c640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19dc1600e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2055097334 0 0) 0x7f19dc0d54f0 con 0x7f19dc0d1510 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.546+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19c4003520 con 0x7f19dc0d1510 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221994137 0 0) 0x7f19dc06d1d0 con 0x7f19dc0d1080 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19dc0d54f0 con 0x7f19dc0d1080 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3403058792 0 0) 0x7f19c4003520 con 0x7f19dc0d1510 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 msgr2=0x7f19dc1600e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19dc1600e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 >> v1:172.21.15.187:6789/0 conn(0x7f19dc0d1080 legacy=0x7f19dc0d3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19dc06d4b0 con 0x7f19dc0d1510 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f19cc001670 con 0x7f19dc0d1510 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 >> v1:172.21.15.150:6789/0 conn(0x7f19dc0d1510 legacy=0x7f19dc0d0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19c4003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c4004d40 0x7f19c4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc06cf80 con 0x7f19c40040d0 2024-04-25T14:02:54.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19c400bbd0 con 0x7f19c4004d40 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19c400bdd0 con 0x7f19dc0ba190 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19dbfff640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19c4003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19e083c640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c4004d40 0x7f19c4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 -- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 msgr2=0x7f19c4003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0ba190 0x7f19c4003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 -- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c4004d40 msgr2=0x7f19c4005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c4004d40 0x7f19c4005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19db7fe640 1 -- 172.21.15.150:0/4170961254 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c4003520 con 0x7f19c40040d0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.547+0000 7f19e083c640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19c4004d40 0x7f19c4005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.548+0000 7f19db7fe640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19c4004800 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f19d00018a0 tx=0x7f19c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b9983bb68d6eba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.548+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d00014f0 con 0x7f19c40040d0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.548+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d00052d0 con 0x7f19c40040d0 2024-04-25T14:02:54.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.548+0000 7f19daffd640 1 -- 172.21.15.150:0/4170961254 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d002ac60 con 0x7f19c40040d0 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 -- 172.21.15.150:0/4170961254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 msgr2=0x7f19c4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 --2- 172.21.15.150:0/4170961254 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19c4004800 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f19d00018a0 tx=0x7f19c4005830 comp rx=0 tx=0).stop 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e083c640 1 -- 172.21.15.150:0/4170961254 reap_dead start 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 -- 172.21.15.150:0/4170961254 shutdown_connections 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 -- 172.21.15.150:0/4170961254 >> 172.21.15.150:0/4170961254 conn(0x7f19dc0b7800 msgr2=0x7f19dc0b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 -- 172.21.15.150:0/4170961254 shutdown_connections 2024-04-25T14:02:54.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.549+0000 7f19e229f640 1 -- 172.21.15.150:0/4170961254 wait complete. 2024-04-25T14:02:54.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.551+0000 7f19e229f640 1 Processor -- start 2024-04-25T14:02:54.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.551+0000 7f19e229f640 1 -- start start 2024-04-25T14:02:54.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0cac60 0x7f19dc162000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19dc0cb070 0x7f19dc162540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc1600e0 con 0x7f19c40040d0 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19dc06da70 con 0x7f19dc0cb070 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e229f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19dc160260 con 0x7f19dc0cac60 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19dbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44854/0 (socket says 172.21.15.150:44854) 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19dbfff640 1 -- 172.21.15.150:0/1841660488 learned_addr learned my addr 172.21.15.150:0/1841660488 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19db7fe640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0cac60 0x7f19dc162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.552+0000 7f19e083c640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19dc0cb070 0x7f19dc162540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 -- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0cac60 msgr2=0x7f19dc162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0cac60 0x7f19dc162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 -- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19dc0cb070 msgr2=0x7f19dc162540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19dc0cb070 0x7f19dc162540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 -- 172.21.15.150:0/1841660488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19dc153830 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19dbfff640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f19cc004f70 tx=0x7f19cc0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc00f750 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19cc018070 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19cc00fb50 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19dc0cb870 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.553+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19dc0cbe30 con 0x7f19c40040d0 2024-04-25T14:02:54.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.554+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19dc0d0370 con 0x7f19c40040d0 2024-04-25T14:02:54.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.555+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f19cc00fcf0 con 0x7f19c40040d0 2024-04-25T14:02:54.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.555+0000 7f19d8ff9640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 0x7f19c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.555+0000 7f19db7fe640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 0x7f19c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.555+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f19cc0954f0 con 0x7f19c40040d0 2024-04-25T14:02:54.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.556+0000 7f19db7fe640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 0x7f19c80769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19d0032000 tx=0x7f19d0009030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.560+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19dc0d0370 con 0x7f19c40040d0 2024-04-25T14:02:54.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.737+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f19dc076a40 con 0x7f19c40040d0 2024-04-25T14:02:54.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.737+0000 7f19d8ff9640 1 -- 172.21.15.150:0/1841660488 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f19dc153830 con 0x7f19c40040d0 2024-04-25T14:02:54.736 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.739+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 msgr2=0x7f19c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.739+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 0x7f19c80769c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19d0032000 tx=0x7f19d0009030 comp rx=0 tx=0).stop 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.739+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 msgr2=0x7f19dc0b39a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.739+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f19cc004f70 tx=0x7f19cc0074f0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 shutdown_connections 2024-04-25T14:02:54.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19c80744b0 0x7f19c80769c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19dc0cb070 0x7f19dc162540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19dc0cac60 0x7f19dc162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 --2- 172.21.15.150:0/1841660488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19c40040d0 0x7f19dc0b39a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 >> 172.21.15.150:0/1841660488 conn(0x7f19dc0b7800 msgr2=0x7f19dc0b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 shutdown_connections 2024-04-25T14:02:54.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.740+0000 7f19e229f640 1 -- 172.21.15.150:0/1841660488 wait complete. 2024-04-25T14:02:54.750 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:54.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.861+0000 7f7ff9d46640 1 Processor -- start 2024-04-25T14:02:54.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.861+0000 7f7ff9d46640 1 -- start start 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff9d46640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ff41628f0 0x7f7ff40b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff9d46640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 0x7f7ff415e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff9d46640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ff40b37b0 con 0x7f7ff40b3a90 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff9d46640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7ff4161e10 con 0x7f7ff41628f0 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff9d46640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ff41620f0 con 0x7f7ff406e130 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ff41628f0 0x7f7ff40b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7ff41628f0 0x7f7ff40b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44860/0 (socket says 172.21.15.150:44860) 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 -- 172.21.15.150:0/1774566004 learned_addr learned my addr 172.21.15.150:0/1774566004 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 0x7f7ff415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 -- 172.21.15.150:0/1774566004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 msgr2=0x7f7ff415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 0x7f7ff415e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 -- 172.21.15.150:0/1774566004 >> v1:172.21.15.187:6789/0 conn(0x7f7ff40b3a90 legacy=0x7f7ff40b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff37fe640 1 -- 172.21.15.150:0/1774566004 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff41623d0 con 0x7f7ff41628f0 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.863+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 0x7f7ff415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.864+0000 7f7ff37fe640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff41628f0 0x7f7ff40b31d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009870 tx=0x7f7fe402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d333c10508bec393 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.864+0000 7f7ff27fc640 1 -- 172.21.15.150:0/1774566004 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4037070 con 0x7f7ff41628f0 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.864+0000 7f7ff27fc640 1 -- 172.21.15.150:0/1774566004 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe4032440 con 0x7f7ff41628f0 2024-04-25T14:02:54.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.864+0000 7f7ff27fc640 1 -- 172.21.15.150:0/1774566004 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4032810 con 0x7f7ff41628f0 2024-04-25T14:02:54.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 -- 172.21.15.150:0/1774566004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff41628f0 msgr2=0x7f7ff40b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff41628f0 0x7f7ff40b31d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009870 tx=0x7f7fe402a210 comp rx=0 tx=0).stop 2024-04-25T14:02:54.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 -- 172.21.15.150:0/1774566004 shutdown_connections 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff406e130 0x7f7ff415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/1774566004 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff41628f0 0x7f7ff40b31d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 -- 172.21.15.150:0/1774566004 >> 172.21.15.150:0/1774566004 conn(0x7f7ff409b4a0 msgr2=0x7f7ff40b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 -- 172.21.15.150:0/1774566004 shutdown_connections 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.865+0000 7f7ff9d46640 1 -- 172.21.15.150:0/1774566004 wait complete. 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.866+0000 7f7ff9d46640 1 Processor -- start 2024-04-25T14:02:54.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.867+0000 7f7ff9d46640 1 -- start start 2024-04-25T14:02:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 0x7f7ff40caef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff41628f0 0x7f7ff40d0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff40b2050 con 0x7f7ff406e130 2024-04-25T14:02:54.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7ff40b1ed0 con 0x7f7ff40b3a90 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff9d46640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ff40b21d0 con 0x7f7ff41628f0 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44866/0 (socket says 172.21.15.150:44866) 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 -- 172.21.15.150:0/3597337252 learned_addr learned my addr 172.21.15.150:0/3597337252 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 0x7f7ff40caef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff3fff640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff41628f0 0x7f7ff40d0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 -- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff41628f0 msgr2=0x7f7ff40d0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff41628f0 0x7f7ff40d0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 -- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 msgr2=0x7f7ff40caef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 0x7f7ff40caef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 -- 172.21.15.150:0/3597337252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ff40c9260 con 0x7f7ff406e130 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 0x7f7ff40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.868+0000 7f7ff37fe640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009870 tx=0x7f7fe40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.869+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4037040 con 0x7f7ff406e130 2024-04-25T14:02:54.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.869+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7fe4042070 con 0x7f7ff406e130 2024-04-25T14:02:54.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.869+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ff40c9540 con 0x7f7ff406e130 2024-04-25T14:02:54.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.869+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7fe4030bb0 con 0x7f7ff406e130 2024-04-25T14:02:54.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.869+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ff40c9b00 con 0x7f7ff406e130 2024-04-25T14:02:54.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.870+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ff4076a40 con 0x7f7ff406e130 2024-04-25T14:02:54.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.870+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7fe4047460 con 0x7f7ff406e130 2024-04-25T14:02:54.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.871+0000 7f7fdbfff640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 0x7f7fe00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:54.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.871+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 0x7f7fe00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:54.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.871+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7fe40b6180 con 0x7f7ff406e130 2024-04-25T14:02:54.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.871+0000 7f7ff2ffd640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 0x7f7fe00769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8006fd0 tx=0x7f7fe8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:54.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:54.875+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7fe407ff50 con 0x7f7ff406e130 2024-04-25T14:02:55.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.051+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7ff40b2f90 con 0x7f7ff406e130 2024-04-25T14:02:55.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.052+0000 7f7fdbfff640 1 -- 172.21.15.150:0/3597337252 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f7fe4084250 con 0x7f7ff406e130 2024-04-25T14:02:55.051 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 msgr2=0x7f7fe00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 0x7f7fe00769c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7fe8006fd0 tx=0x7f7fe8008040 comp rx=0 tx=0).stop 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 msgr2=0x7f7ff4155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4009870 tx=0x7f7fe40046c0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 shutdown_connections 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7fe00744b0 0x7f7fe00769c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ff41628f0 0x7f7ff40d0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ff40b3a90 0x7f7ff40caef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 --2- 172.21.15.150:0/3597337252 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ff406e130 0x7f7ff4155270 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 >> 172.21.15.150:0/3597337252 conn(0x7f7ff409b4a0 msgr2=0x7f7ff4071460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 shutdown_connections 2024-04-25T14:02:55.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.054+0000 7f7ff9d46640 1 -- 172.21.15.150:0/3597337252 wait complete. 2024-04-25T14:02:55.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:55.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.175+0000 7f0f90040640 1 Processor -- start 2024-04-25T14:02:55.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.175+0000 7f0f90040640 1 -- start start 2024-04-25T14:02:55.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.176+0000 7f0f90040640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f880735b0 0x7f0f88160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.176+0000 7f0f90040640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f88164e10 con 0x7f0f880cf330 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.176+0000 7f0f90040640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f880b6c70 con 0x7f0f880d3290 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.176+0000 7f0f90040640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0f88160770 con 0x7f0f880735b0 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8d5db640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0f880735b0 0x7f0f88160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8e5dd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f0f880cf330 0x7f0f880ce000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55226/0 (socket says 172.21.15.150:55226) 2024-04-25T14:02:55.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8e5dd640 1 -- 172.21.15.150:0/2119640930 learned_addr learned my addr 172.21.15.150:0/2119640930 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428695521 0 0) 0x7f0f88164e10 con 0x7f0f880cf330 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f74003520 con 0x7f0f880cf330 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8d5db640 1 -- 172.21.15.150:0/2119640930 >> v1:172.21.15.187:6789/0 conn(0x7f0f880d3290 legacy=0x7f0f88072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2472730920 0 0) 0x7f0f880b6c70 con 0x7f0f880d3290 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8d5db640 1 -- 172.21.15.150:0/2119640930 >> v1:172.21.15.150:6789/0 conn(0x7f0f880cf330 legacy=0x7f0f880ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.177+0000 7f0f8d5db640 1 -- 172.21.15.150:0/2119640930 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f88164050 con 0x7f0f880735b0 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.178+0000 7f0f8d5db640 1 --2- 172.21.15.150:0/2119640930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f880735b0 0x7f0f88160230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f780097a0 tx=0x7f0f7802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6efabdd4e3a098a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.178+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78037070 con 0x7f0f880735b0 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.178+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f78032440 con 0x7f0f880735b0 2024-04-25T14:02:55.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.178+0000 7f0f8cdda640 1 -- 172.21.15.150:0/2119640930 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78032810 con 0x7f0f880735b0 2024-04-25T14:02:55.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 -- 172.21.15.150:0/2119640930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f880735b0 msgr2=0x7f0f88160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 --2- 172.21.15.150:0/2119640930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f880735b0 0x7f0f88160230 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0f780097a0 tx=0x7f0f7802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 -- 172.21.15.150:0/2119640930 shutdown_connections 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 --2- 172.21.15.150:0/2119640930 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f880735b0 0x7f0f88160230 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 -- 172.21.15.150:0/2119640930 >> 172.21.15.150:0/2119640930 conn(0x7f0f880d39b0 msgr2=0x7f0f880d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 -- 172.21.15.150:0/2119640930 shutdown_connections 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.179+0000 7f0f90040640 1 -- 172.21.15.150:0/2119640930 wait complete. 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.180+0000 7f0f90040640 1 Processor -- start 2024-04-25T14:02:55.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.181+0000 7f0f90040640 1 -- start start 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f880cf330 0x7f0f88149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f8815f5c0 0x7f0f8815e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0f881648f0 con 0x7f0f8815ee40 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0f88164610 con 0x7f0f8815f5c0 2024-04-25T14:02:55.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f90040640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0f880b6800 con 0x7f0f880cf330 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44890/0 (socket says 172.21.15.150:44890) 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 -- 172.21.15.150:0/1267772562 learned_addr learned my addr 172.21.15.150:0/1267772562 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8d5db640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f8815f5c0 0x7f0f8815e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8dddc640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f880cf330 0x7f0f88149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 -- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f880cf330 msgr2=0x7f0f88149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f880cf330 0x7f0f88149eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 -- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f8815f5c0 msgr2=0x7f0f8815e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f8815f5c0 0x7f0f8815e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 -- 172.21.15.150:0/1267772562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8815f250 con 0x7f0f8815ee40 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.182+0000 7f0f8e5dd640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0f80001eb0 tx=0x7f0f8000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.183+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f80017070 con 0x7f0f8815ee40 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.183+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f80005ce0 con 0x7f0f8815ee40 2024-04-25T14:02:55.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.183+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8806e4f0 con 0x7f0f8815ee40 2024-04-25T14:02:55.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.183+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f800072d0 con 0x7f0f8815ee40 2024-04-25T14:02:55.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.183+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8806eab0 con 0x7f0f8815ee40 2024-04-25T14:02:55.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.184+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f88076a40 con 0x7f0f8815ee40 2024-04-25T14:02:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.184+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0f8001a070 con 0x7f0f8815ee40 2024-04-25T14:02:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.185+0000 7f0f7e7fc640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 0x7f0f6c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.185+0000 7f0f8dddc640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 0x7f0f6c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.185+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0f80094930 con 0x7f0f8815ee40 2024-04-25T14:02:55.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.185+0000 7f0f8dddc640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 0x7f0f6c0769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f840012e0 tx=0x7f0f84009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.190+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f8815f250 con 0x7f0f8815ee40 2024-04-25T14:02:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.367+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0f880b94a0 con 0x7f0f8815ee40 2024-04-25T14:02:55.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.367+0000 7f0f7e7fc640 1 -- 172.21.15.150:0/1267772562 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f0f8005e670 con 0x7f0f8815ee40 2024-04-25T14:02:55.366 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 msgr2=0x7f0f6c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 0x7f0f6c0769c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f840012e0 tx=0x7f0f84009040 comp rx=0 tx=0).stop 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 msgr2=0x7f0f880b52d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0f80001eb0 tx=0x7f0f8000c820 comp rx=0 tx=0).stop 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 shutdown_connections 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0f6c0744b0 0x7f0f6c0769c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0f8815ee40 0x7f0f880b52d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0f8815f5c0 0x7f0f8815e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 --2- 172.21.15.150:0/1267772562 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0f880cf330 0x7f0f88149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 >> 172.21.15.150:0/1267772562 conn(0x7f0f880d39b0 msgr2=0x7f0f880b81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 shutdown_connections 2024-04-25T14:02:55.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.369+0000 7f0f90040640 1 -- 172.21.15.150:0/1267772562 wait complete. 2024-04-25T14:02:55.380 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:55.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.489+0000 7fadce242640 1 Processor -- start 2024-04-25T14:02:55.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.489+0000 7fadce242640 1 -- start start 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadce242640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 0x7fadc8160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadce242640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fadc80b78b0 0x7fadc806cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadce242640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadc80b2de0 con 0x7fadc80b74a0 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadce242640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fadc80b39b0 con 0x7fadc80b78b0 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadce242640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fadc80b4550 con 0x7fadc80b2650 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadc6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fadc80b74a0 0x7fadc80b3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55230/0 (socket says 172.21.15.150:55230) 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.490+0000 7fadc6ffd640 1 -- 172.21.15.150:0/2229047772 learned_addr learned my addr 172.21.15.150:0/2229047772 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:55.490 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc7fff640 1 --2- 172.21.15.150:0/2229047772 >> v2:172.21.15.187:3300/0 conn(0x7fadc80b78b0 0x7fadc806cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc77fe640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 0x7fadc8160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc67fc640 1 -- 172.21.15.150:0/2229047772 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2665572857 0 0) 0x7fadc80b2de0 con 0x7fadc80b74a0 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc67fc640 1 -- 172.21.15.150:0/2229047772 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fadb0003520 con 0x7fadc80b74a0 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc7fff640 1 -- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 msgr2=0x7fadc8160160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc7fff640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 0x7fadc8160160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc7fff640 1 -- 172.21.15.150:0/2229047772 >> v1:172.21.15.150:6789/0 conn(0x7fadc80b74a0 legacy=0x7fadc80b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc7fff640 1 -- 172.21.15.150:0/2229047772 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc80b3080 con 0x7fadc80b78b0 2024-04-25T14:02:55.491 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.491+0000 7fadc77fe640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 0x7fadc8160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:02:55.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.492+0000 7fadc7fff640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc806cbf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009d90 tx=0x7fadc002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32a4f1ee8b7bb1b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.492+0000 7fadc67fc640 1 -- 172.21.15.150:0/2229047772 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0007740 con 0x7fadc80b78b0 2024-04-25T14:02:55.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.492+0000 7fadc67fc640 1 -- 172.21.15.150:0/2229047772 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadc00078e0 con 0x7fadc80b78b0 2024-04-25T14:02:55.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.492+0000 7fadc67fc640 1 -- 172.21.15.150:0/2229047772 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0007cb0 con 0x7fadc80b78b0 2024-04-25T14:02:55.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 -- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 msgr2=0x7fadc806cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc806cbf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fadc0009d90 tx=0x7fadc002a5f0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 -- 172.21.15.150:0/2229047772 shutdown_connections 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc806cbf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 --2- 172.21.15.150:0/2229047772 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b2650 0x7fadc8160160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 -- 172.21.15.150:0/2229047772 >> 172.21.15.150:0/2229047772 conn(0x7fadc8073ce0 msgr2=0x7fadc809b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 -- 172.21.15.150:0/2229047772 shutdown_connections 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.493+0000 7fadce242640 1 -- 172.21.15.150:0/2229047772 wait complete. 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.494+0000 7fadce242640 1 Processor -- start 2024-04-25T14:02:55.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.495+0000 7fadce242640 1 -- start start 2024-04-25T14:02:55.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b74a0 0x7fadc81f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc81fa260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fadc806d130 con 0x7fadc80b2650 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fadc80b3640 con 0x7fadc80b78b0 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadce242640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fadc806d2b0 con 0x7fadc80b74a0 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44894/0 (socket says 172.21.15.150:44894) 2024-04-25T14:02:55.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 -- 172.21.15.150:0/3566849466 learned_addr learned my addr 172.21.15.150:0/3566849466 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 -- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b74a0 msgr2=0x7fadc81f9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b74a0 0x7fadc81f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 -- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 msgr2=0x7fadc81fa260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc81fa260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 -- 172.21.15.150:0/3566849466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc8161a50 con 0x7fadc80b2650 2024-04-25T14:02:55.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadc77fe640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fadb800e9c0 tx=0x7fadb800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.496+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb8017070 con 0x7fadc80b2650 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.497+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadb800ca00 con 0x7fadc80b2650 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.497+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc8161d30 con 0x7fadc80b2650 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.497+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb800cdb0 con 0x7fadc80b2650 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.497+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc8146430 con 0x7fadc80b2650 2024-04-25T14:02:55.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.497+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadc8076a40 con 0x7fadc80b2650 2024-04-25T14:02:55.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.498+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fadb801a070 con 0x7fadc80b2650 2024-04-25T14:02:55.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.498+0000 7fadaffff640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 0x7fadb40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.499+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fadb80956e0 con 0x7fadc80b2650 2024-04-25T14:02:55.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.499+0000 7fadc6ffd640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 0x7fadb40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.499+0000 7fadc6ffd640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 0x7fadb40769f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadc80d2f20 tx=0x7fadbc0078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.503+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadb805fb90 con 0x7fadc80b2650 2024-04-25T14:02:55.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.677+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadc8163570 con 0x7fadc80b2650 2024-04-25T14:02:55.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.679+0000 7fadaffff640 1 -- 172.21.15.150:0/3566849466 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fadb805f490 con 0x7fadc80b2650 2024-04-25T14:02:55.678 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:55.678 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 msgr2=0x7fadb40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 0x7fadb40769f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fadc80d2f20 tx=0x7fadbc0078f0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 msgr2=0x7fadc8160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fadb800e9c0 tx=0x7fadb800ee80 comp rx=0 tx=0).stop 2024-04-25T14:02:55.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 shutdown_connections 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fadb40744e0 0x7fadb40769f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fadc80b78b0 0x7fadc81fa260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fadc80b74a0 0x7fadc81f9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 --2- 172.21.15.150:0/3566849466 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fadc80b2650 0x7fadc8160fb0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 >> 172.21.15.150:0/3566849466 conn(0x7fadc8073ce0 msgr2=0x7fadc80b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 shutdown_connections 2024-04-25T14:02:55.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.681+0000 7fadce242640 1 -- 172.21.15.150:0/3566849466 wait complete. 2024-04-25T14:02:55.692 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:55.692 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:55.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.803+0000 7febabaf0640 1 Processor -- start 2024-04-25T14:02:55.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.803+0000 7febabaf0640 1 -- start start 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feba40d3290 0x7feba4160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 0x7feba4072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feba40733e0 0x7feba40d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7feba4160770 con 0x7feba40733e0 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7feba40737f0 con 0x7feba40d3290 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febabaf0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feba4164e10 con 0x7feba40cf330 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febaa08d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feba40733e0 0x7feba40d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febaa08d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feba40733e0 0x7feba40d18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44908/0 (socket says 172.21.15.150:44908) 2024-04-25T14:02:55.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.804+0000 7febaa08d640 1 -- 172.21.15.150:0/2169833615 learned_addr learned my addr 172.21.15.150:0/2169833615 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7feba988c640 1 --2- 172.21.15.150:0/2169833615 >> v2:172.21.15.187:3300/0 conn(0x7feba40d3290 0x7feba4160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7feba908b640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 0x7feba4072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 -- 172.21.15.150:0/2169833615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 msgr2=0x7feba4072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 0x7feba4072ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 -- 172.21.15.150:0/2169833615 >> v2:172.21.15.187:3300/0 conn(0x7feba40d3290 msgr2=0x7feba4160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 --2- 172.21.15.150:0/2169833615 >> v2:172.21.15.187:3300/0 conn(0x7feba40d3290 0x7feba4160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 -- 172.21.15.150:0/2169833615 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba40b6c70 con 0x7feba40733e0 2024-04-25T14:02:55.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7feba908b640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 0x7feba4072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.805+0000 7febaa08d640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40733e0 0x7feba40d18b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feba408cc30 tx=0x7feb9c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6828e1d88764c52b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7feba888a640 1 -- 172.21.15.150:0/2169833615 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c036070 con 0x7feba40733e0 2024-04-25T14:02:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7feba888a640 1 -- 172.21.15.150:0/2169833615 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c031600 con 0x7feba40733e0 2024-04-25T14:02:55.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7feba888a640 1 -- 172.21.15.150:0/2169833615 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0319d0 con 0x7feba40733e0 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 -- 172.21.15.150:0/2169833615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40733e0 msgr2=0x7feba40d18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40733e0 0x7feba40d18b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7feba408cc30 tx=0x7feb9c02a610 comp rx=0 tx=0).stop 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 -- 172.21.15.150:0/2169833615 shutdown_connections 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba40733e0 0x7feba40d18b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 --2- 172.21.15.150:0/2169833615 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40cf330 0x7feba4072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 --2- 172.21.15.150:0/2169833615 >> v2:172.21.15.187:3300/0 conn(0x7feba40d3290 0x7feba4160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 -- 172.21.15.150:0/2169833615 >> 172.21.15.150:0/2169833615 conn(0x7feba40d39b0 msgr2=0x7feba40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 -- 172.21.15.150:0/2169833615 shutdown_connections 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.806+0000 7febabaf0640 1 -- 172.21.15.150:0/2169833615 wait complete. 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.808+0000 7febabaf0640 1 Processor -- start 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.808+0000 7febabaf0640 1 -- start start 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40733e0 0x7feba4149880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 0x7feba415c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feba4164610 con 0x7feba41f6270 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feba4164330 con 0x7feba40cf330 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.809+0000 7febabaf0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feba41648f0 con 0x7feba40733e0 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44922/0 (socket says 172.21.15.150:44922) 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 -- 172.21.15.150:0/4248237459 learned_addr learned my addr 172.21.15.150:0/4248237459 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feba988c640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40733e0 0x7feba4149880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feba908b640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 0x7feba415c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 -- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40733e0 msgr2=0x7feba4149880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40733e0 0x7feba4149880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 -- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 msgr2=0x7feba415c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 0x7feba415c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 -- 172.21.15.150:0/4248237459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba406f1b0 con 0x7feba41f6270 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feba908b640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 0x7feba415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febaa08d640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009f40 tx=0x7feb9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c036050 con 0x7feba41f6270 2024-04-25T14:02:55.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c042070 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba406f490 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c009510 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.810+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba406fa50 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.811+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba4076a40 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.812+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feb9c007bd0 con 0x7feba41f6270 2024-04-25T14:02:55.814 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.812+0000 7feb9a7fc640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 0x7feb880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:55.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.812+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7feb9c0b6470 con 0x7feba41f6270 2024-04-25T14:02:55.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.812+0000 7feba988c640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 0x7feb880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:55.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.813+0000 7feba988c640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 0x7feb880769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feba00079a0 tx=0x7feba0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:55.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.817+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feb9c080800 con 0x7feba41f6270 2024-04-25T14:02:55.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.991+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7feba40d3000 con 0x7feba41f6270 2024-04-25T14:02:55.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.991+0000 7feb9a7fc640 1 -- 172.21.15.150:0/4248237459 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7feb9c080100 con 0x7feba41f6270 2024-04-25T14:02:55.991 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 msgr2=0x7feb880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 0x7feb880769c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7feba00079a0 tx=0x7feba0008040 comp rx=0 tx=0).stop 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 msgr2=0x7feba4070250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7feb9c009f40 tx=0x7feb9c004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 shutdown_connections 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feb880744b0 0x7feb880769c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feba41f6270 0x7feba4070250 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feba40cf330 0x7feba415c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 --2- 172.21.15.150:0/4248237459 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feba40733e0 0x7feba4149880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:55.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.993+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 >> 172.21.15.150:0/4248237459 conn(0x7feba40d39b0 msgr2=0x7feba4162420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:55.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.994+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 shutdown_connections 2024-04-25T14:02:55.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:55.994+0000 7febabaf0640 1 -- 172.21.15.150:0/4248237459 wait complete. 2024-04-25T14:02:56.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:56.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.115+0000 7fe9e0225640 1 Processor -- start 2024-04-25T14:02:56.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.115+0000 7fe9e0225640 1 -- start start 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9e0225640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9d81628f0 0x7fe9d80b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9e0225640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe9d80b41a0 0x7fe9d815e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9e0225640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9d80b37b0 con 0x7fe9d806e130 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9e0225640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d8161e10 con 0x7fe9d81628f0 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9e0225640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe9d81620f0 con 0x7fe9d80b41a0 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9d81628f0 0x7fe9d80b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe9d81628f0 0x7fe9d80b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44926/0 (socket says 172.21.15.150:44926) 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/3619641459 learned_addr learned my addr 172.21.15.150:0/3619641459 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9de7c2640 1 --2- 172.21.15.150:0/3619641459 >> v2:172.21.15.187:3300/0 conn(0x7fe9d80b41a0 0x7fe9d815e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/3619641459 >> v2:172.21.15.187:3300/0 conn(0x7fe9d80b41a0 msgr2=0x7fe9d815e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:56.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 --2- 172.21.15.150:0/3619641459 >> v2:172.21.15.187:3300/0 conn(0x7fe9d80b41a0 0x7fe9d815e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/3619641459 >> v1:172.21.15.187:6789/0 conn(0x7fe9d806e130 legacy=0x7fe9d80b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:56.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.117+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/3619641459 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d81623d0 con 0x7fe9d81628f0 2024-04-25T14:02:56.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.118+0000 7fe9ddfc1640 1 --2- 172.21.15.150:0/3619641459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d81628f0 0x7fe9d80b31d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe9d806e0c0 tx=0x7fe9d402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e5513f6ae0596a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.118+0000 7fe9dcfbf640 1 -- 172.21.15.150:0/3619641459 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4036070 con 0x7fe9d81628f0 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.118+0000 7fe9dcfbf640 1 -- 172.21.15.150:0/3619641459 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d4031600 con 0x7fe9d81628f0 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.118+0000 7fe9dcfbf640 1 -- 172.21.15.150:0/3619641459 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d40319d0 con 0x7fe9d81628f0 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 -- 172.21.15.150:0/3619641459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d81628f0 msgr2=0x7fe9d80b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 --2- 172.21.15.150:0/3619641459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d81628f0 0x7fe9d80b31d0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fe9d806e0c0 tx=0x7fe9d402a6f0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 -- 172.21.15.150:0/3619641459 shutdown_connections 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 --2- 172.21.15.150:0/3619641459 >> v2:172.21.15.187:3300/0 conn(0x7fe9d80b41a0 0x7fe9d815e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 --2- 172.21.15.150:0/3619641459 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d81628f0 0x7fe9d80b31d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 -- 172.21.15.150:0/3619641459 >> 172.21.15.150:0/3619641459 conn(0x7fe9d809b4a0 msgr2=0x7fe9d80b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 -- 172.21.15.150:0/3619641459 shutdown_connections 2024-04-25T14:02:56.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.119+0000 7fe9e0225640 1 -- 172.21.15.150:0/3619641459 wait complete. 2024-04-25T14:02:56.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.121+0000 7fe9e0225640 1 Processor -- start 2024-04-25T14:02:56.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.121+0000 7fe9e0225640 1 -- start start 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 0x7fe9d81faa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9d81628f0 0x7fe9d81f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d80b2050 con 0x7fe9d806e130 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe9d80b1ed0 con 0x7fe9d80b41a0 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9e0225640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe9d80b21d0 con 0x7fe9d81628f0 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44942/0 (socket says 172.21.15.150:44942) 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/1221827796 learned_addr learned my addr 172.21.15.150:0/1221827796 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9dd7c0640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 0x7fe9d81faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9de7c2640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9d81628f0 0x7fe9d81f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9d81628f0 msgr2=0x7fe9d81f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9d81628f0 0x7fe9d81f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 msgr2=0x7fe9d81faa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 0x7fe9d81faa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9ddfc1640 1 -- 172.21.15.150:0/1221827796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9d80d0d20 con 0x7fe9d806e130 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.122+0000 7fe9dd7c0640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 0x7fe9d81faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9ddfc1640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4034040 tx=0x7fe9d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4036050 con 0x7fe9d806e130 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9d4042070 con 0x7fe9d806e130 2024-04-25T14:02:56.124 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9d80d0f90 con 0x7fe9d806e130 2024-04-25T14:02:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4009510 con 0x7fe9d806e130 2024-04-25T14:02:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.123+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9d80c9e30 con 0x7fe9d806e130 2024-04-25T14:02:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.124+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9d8076a40 con 0x7fe9d806e130 2024-04-25T14:02:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.125+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe9d4007c50 con 0x7fe9d806e130 2024-04-25T14:02:56.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.125+0000 7fe9ce7fc640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 0x7fe9bc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.125+0000 7fe9dd7c0640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 0x7fe9bc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.125+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe9d40b6fb0 con 0x7fe9d806e130 2024-04-25T14:02:56.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.126+0000 7fe9dd7c0640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 0x7fe9bc0769f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8006fd0 tx=0x7fe9c8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.129+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe9d4080e60 con 0x7fe9d806e130 2024-04-25T14:02:56.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.303+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe9d80b2f90 con 0x7fe9d806e130 2024-04-25T14:02:56.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.304+0000 7fe9ce7fc640 1 -- 172.21.15.150:0/1221827796 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fe9d4085160 con 0x7fe9d806e130 2024-04-25T14:02:56.303 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 msgr2=0x7fe9bc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 0x7fe9bc0769f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe9c8006fd0 tx=0x7fe9c8008040 comp rx=0 tx=0).stop 2024-04-25T14:02:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 msgr2=0x7fe9d80ca950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.305 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4034040 tx=0x7fe9d4004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 shutdown_connections 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe9bc0744e0 0x7fe9bc0769f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe9d81628f0 0x7fe9d81f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe9d80b41a0 0x7fe9d81faa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 --2- 172.21.15.150:0/1221827796 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe9d806e130 0x7fe9d80ca950 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 >> 172.21.15.150:0/1221827796 conn(0x7fe9d809b4a0 msgr2=0x7fe9d815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 shutdown_connections 2024-04-25T14:02:56.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.306+0000 7fe9e0225640 1 -- 172.21.15.150:0/1221827796 wait complete. 2024-04-25T14:02:56.316 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:56.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.428+0000 7f72c7ae0640 1 Processor -- start 2024-04-25T14:02:56.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.428+0000 7f72c7ae0640 1 -- start start 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.429+0000 7f72c7ae0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72c00b2b10 0x7f72c015e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.429+0000 7f72c7ae0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 0x7f72c00b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.429+0000 7f72c7ae0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72c00b5d10 con 0x7f72c00d3260 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.429+0000 7f72c7ae0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f72c00d2dd0 con 0x7f72c00b2b10 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.429+0000 7f72c7ae0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72c00d21c0 con 0x7f72c00b53c0 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72c00b2b10 0x7f72c015e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72c00b2b10 0x7f72c015e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44948/0 (socket says 172.21.15.150:44948) 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c507b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f72c00d3260 0x7f72c00cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55244/0 (socket says 172.21.15.150:55244) 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 -- 172.21.15.150:0/366599163 learned_addr learned my addr 172.21.15.150:0/366599163 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c607d640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 0x7f72c00b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 -- 172.21.15.150:0/366599163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 msgr2=0x7f72c00b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 0x7f72c00b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 -- 172.21.15.150:0/366599163 >> v1:172.21.15.150:6789/0 conn(0x7f72c00d3260 legacy=0x7f72c00cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c487a640 1 -- 172.21.15.150:0/366599163 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727534734 0 0) 0x7f72b0001110 con 0x7f72c00d3260 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c587c640 1 -- 172.21.15.150:0/366599163 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c00d24a0 con 0x7f72c00b2b10 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.430+0000 7f72c607d640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 0x7f72c00b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c587c640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00b2b10 0x7f72c015e010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009870 tx=0x7f72bc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7247a9a5b4bed55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c487a640 1 -- 172.21.15.150:0/366599163 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc037070 con 0x7f72c00b2b10 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c487a640 1 -- 172.21.15.150:0/366599163 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc032440 con 0x7f72c00b2b10 2024-04-25T14:02:56.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c487a640 1 -- 172.21.15.150:0/366599163 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc032810 con 0x7f72c00b2b10 2024-04-25T14:02:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 -- 172.21.15.150:0/366599163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00b2b10 msgr2=0x7f72c015e010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00b2b10 0x7f72c015e010 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009870 tx=0x7f72bc02a270 comp rx=0 tx=0).stop 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 -- 172.21.15.150:0/366599163 shutdown_connections 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b53c0 0x7f72c00b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/366599163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00b2b10 0x7f72c015e010 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.431+0000 7f72c7ae0640 1 -- 172.21.15.150:0/366599163 >> 172.21.15.150:0/366599163 conn(0x7f72c00d03c0 msgr2=0x7f72c00d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.432+0000 7f72c7ae0640 1 -- 172.21.15.150:0/366599163 shutdown_connections 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.432+0000 7f72c7ae0640 1 -- 172.21.15.150:0/366599163 wait complete. 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.433+0000 7f72c7ae0640 1 Processor -- start 2024-04-25T14:02:56.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.433+0000 7f72c7ae0640 1 -- start start 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b2b10 0x7f72c0159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c00b53c0 0x7f72c00cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f72c00d3670 con 0x7f72c00d3260 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f72c00d2a60 con 0x7f72c00b53c0 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72c00b6d30 con 0x7f72c00b2b10 2024-04-25T14:02:56.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44950/0 (socket says 172.21.15.150:44950) 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 -- 172.21.15.150:0/3977516204 learned_addr learned my addr 172.21.15.150:0/3977516204 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c587c640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b2b10 0x7f72c0159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c507b640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c00b53c0 0x7f72c00cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 -- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b2b10 msgr2=0x7f72c0159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b2b10 0x7f72c0159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 -- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c00b53c0 msgr2=0x7f72c00cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c00b53c0 0x7f72c00cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 -- 172.21.15.150:0/3977516204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c015a3f0 con 0x7f72c00d3260 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c607d640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72b800cd80 tx=0x7f72b800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b800f680 con 0x7f72c00d3260 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72b800f840 con 0x7f72c00d3260 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.435+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c00ca870 con 0x7f72c00d3260 2024-04-25T14:02:56.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.436+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b800fc30 con 0x7f72c00d3260 2024-04-25T14:02:56.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.436+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c00cae30 con 0x7f72c00d3260 2024-04-25T14:02:56.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.436+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c0076a40 con 0x7f72c00d3260 2024-04-25T14:02:56.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.437+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f72b80213f0 con 0x7f72c00d3260 2024-04-25T14:02:56.437 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.437+0000 7f72b67fc640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 0x7f72a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.438+0000 7f72c587c640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 0x7f72a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.438+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f72b80961e0 con 0x7f72c00d3260 2024-04-25T14:02:56.438 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.438+0000 7f72c587c640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 0x7f72a40769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f72bc034000 tx=0x7f72bc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.442+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72c015a3f0 con 0x7f72c00d3260 2024-04-25T14:02:56.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.617+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f72c00cd750 con 0x7f72c00d3260 2024-04-25T14:02:56.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.618+0000 7f72b67fc640 1 -- 172.21.15.150:0/3977516204 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f72b8014030 con 0x7f72c00d3260 2024-04-25T14:02:56.617 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:56.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 msgr2=0x7f72a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 0x7f72a40769c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f72bc034000 tx=0x7f72bc02e000 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 msgr2=0x7f72c01f3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72b800cd80 tx=0x7f72b800a650 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 shutdown_connections 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72a40744b0 0x7f72a40769c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72c00d3260 0x7f72c01f3f60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72c00b53c0 0x7f72c00cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 --2- 172.21.15.150:0/3977516204 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72c00b2b10 0x7f72c0159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 >> 172.21.15.150:0/3977516204 conn(0x7f72c00d03c0 msgr2=0x7f72c015d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 shutdown_connections 2024-04-25T14:02:56.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.620+0000 7f72c7ae0640 1 -- 172.21.15.150:0/3977516204 wait complete. 2024-04-25T14:02:56.631 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.742+0000 7f605b214640 1 Processor -- start 2024-04-25T14:02:56.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.742+0000 7f605b214640 1 -- start start 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f605b214640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6054160770 0x7f605415a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f605b214640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6054160ce0 con 0x7f605409bbb0 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f605b214640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6054160fc0 con 0x7f605406f2d0 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f605b214640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f60541612a0 con 0x7f6054160770 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f6058fb0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6054160770 0x7f605415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f60597b1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f605409bbb0 0x7f60540b3070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55250/0 (socket says 172.21.15.150:55250) 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f60597b1640 1 -- 172.21.15.150:0/2848829666 learned_addr learned my addr 172.21.15.150:0/2848829666 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:56.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f6058fb0640 1 -- 172.21.15.150:0/2848829666 >> v1:172.21.15.187:6789/0 conn(0x7f605406f2d0 legacy=0x7f605409b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f6058fb0640 1 -- 172.21.15.150:0/2848829666 >> v1:172.21.15.150:6789/0 conn(0x7f605409bbb0 legacy=0x7f60540b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f6058fb0640 1 -- 172.21.15.150:0/2848829666 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6054161580 con 0x7f6054160770 2024-04-25T14:02:56.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.744+0000 7f60537fe640 1 -- 172.21.15.150:0/2848829666 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034596329 0 0) 0x7f6054160ce0 con 0x7f605409bbb0 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.745+0000 7f6058fb0640 1 --2- 172.21.15.150:0/2848829666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f605415a850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60440097a0 tx=0x7f604402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f08802d2a0c65135 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.745+0000 7f60537fe640 1 -- 172.21.15.150:0/2848829666 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044037070 con 0x7f6054160770 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.745+0000 7f60537fe640 1 -- 172.21.15.150:0/2848829666 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6044032440 con 0x7f6054160770 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.745+0000 7f60537fe640 1 -- 172.21.15.150:0/2848829666 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6044032810 con 0x7f6054160770 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 -- 172.21.15.150:0/2848829666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 msgr2=0x7f605415a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 --2- 172.21.15.150:0/2848829666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f605415a850 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60440097a0 tx=0x7f604402a230 comp rx=0 tx=0).stop 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 -- 172.21.15.150:0/2848829666 shutdown_connections 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 --2- 172.21.15.150:0/2848829666 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f605415a850 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 -- 172.21.15.150:0/2848829666 >> 172.21.15.150:0/2848829666 conn(0x7f60540b7680 msgr2=0x7f60540b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 -- 172.21.15.150:0/2848829666 shutdown_connections 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.746+0000 7f605b214640 1 -- 172.21.15.150:0/2848829666 wait complete. 2024-04-25T14:02:56.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.747+0000 7f605b214640 1 Processor -- start 2024-04-25T14:02:56.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.747+0000 7f605b214640 1 -- start start 2024-04-25T14:02:56.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605406f2d0 0x7f605409d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6054145740 0x7f6054144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f60540b3900 con 0x7f6054160770 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f60540b3780 con 0x7f6054145740 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f605b214640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f60540b3a80 con 0x7f605406f2d0 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44968/0 (socket says 172.21.15.150:44968) 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 -- 172.21.15.150:0/837267220 learned_addr learned my addr 172.21.15.150:0/837267220 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f60597b1640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6054145740 0x7f6054144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6058fb0640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605406f2d0 0x7f605409d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 -- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605406f2d0 msgr2=0x7f605409d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605406f2d0 0x7f605409d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 -- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6054145740 msgr2=0x7f6054144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6054145740 0x7f6054144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.749+0000 7f6053fff640 1 -- 172.21.15.150:0/837267220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6054145ce0 con 0x7f6054160770 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f6053fff640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f604800c3f0 tx=0x7f604800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048017070 con 0x7f6054160770 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60480052d0 con 0x7f6054160770 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6054145fc0 con 0x7f6054160770 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6048003830 con 0x7f6054160770 2024-04-25T14:02:56.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.750+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6054146580 con 0x7f6054160770 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.751+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6054076a40 con 0x7f6054160770 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.751+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f604801a070 con 0x7f6054160770 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.752+0000 7f60517fa640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 0x7f60400769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.752+0000 7f6058fb0640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 0x7f60400769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.752+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6048094840 con 0x7f6054160770 2024-04-25T14:02:56.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.752+0000 7f6058fb0640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 0x7f60400769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6044034000 tx=0x7f604402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:56.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.757+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f604805e580 con 0x7f6054160770 2024-04-25T14:02:56.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.930+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f60540b2600 con 0x7f6054160770 2024-04-25T14:02:56.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.931+0000 7f60517fa640 1 -- 172.21.15.150:0/837267220 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f6048062880 con 0x7f6054160770 2024-04-25T14:02:56.931 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:56.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 msgr2=0x7f60400769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 0x7f60400769c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6044034000 tx=0x7f604402e000 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 msgr2=0x7f60540a02b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f604800c3f0 tx=0x7f604800c8b0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 shutdown_connections 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f60400744b0 0x7f60400769c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6054145740 0x7f6054144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6054160770 0x7f60540a02b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 --2- 172.21.15.150:0/837267220 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f605406f2d0 0x7f605409d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 >> 172.21.15.150:0/837267220 conn(0x7f60540b7680 msgr2=0x7f60540afae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 shutdown_connections 2024-04-25T14:02:56.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:56.933+0000 7f605b214640 1 -- 172.21.15.150:0/837267220 wait complete. 2024-04-25T14:02:56.972 INFO:tasks.cephfs_test_runner:Starting test: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) The maximum number of openfiles omap objects keys are now equal to 2024-04-25T14:02:56.973 INFO:tasks.cephfs_test_runner:test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) 2024-04-25T14:02:56.973 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:02:57.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.082+0000 7fd966134640 1 Processor -- start 2024-04-25T14:02:57.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.082+0000 7fd966134640 1 -- start start 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 0x7fd96006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 0x7fd9600b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9600b5d30 0x7fd9600ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd96015e310 con 0x7fd9600b5d30 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd9600b5360 con 0x7fd96006f780 2024-04-25T14:02:57.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd966134640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9600b6140 con 0x7fd96015df00 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9600b5d30 0x7fd9600ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd9600b5d30 0x7fd9600ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44976/0 (socket says 172.21.15.150:44976) 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 -- 172.21.15.150:0/4211354740 learned_addr learned my addr 172.21.15.150:0/4211354740 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95effd640 1 --2- 172.21.15.150:0/4211354740 >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 0x7fd9600b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95f7fe640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 0x7fd96006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 -- 172.21.15.150:0/4211354740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 msgr2=0x7fd96006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 0x7fd96006f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 -- 172.21.15.150:0/4211354740 >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 msgr2=0x7fd9600b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 --2- 172.21.15.150:0/4211354740 >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 0x7fd9600b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95ffff640 1 -- 172.21.15.150:0/4211354740 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9600b4750 con 0x7fd9600b5d30 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95effd640 1 --2- 172.21.15.150:0/4211354740 >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 0x7fd9600b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.084+0000 7fd95f7fe640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 0x7fd96006f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.085+0000 7fd95ffff640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9600b5d30 0x7fd9600ae7e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd960074530 tx=0x7fd95802c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb58a90f062658b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.085+0000 7fd95e7fc640 1 -- 172.21.15.150:0/4211354740 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958039070 con 0x7fd9600b5d30 2024-04-25T14:02:57.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.085+0000 7fd95e7fc640 1 -- 172.21.15.150:0/4211354740 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd958009600 con 0x7fd9600b5d30 2024-04-25T14:02:57.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.085+0000 7fd95e7fc640 1 -- 172.21.15.150:0/4211354740 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9580099f0 con 0x7fd9600b5d30 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 -- 172.21.15.150:0/4211354740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9600b5d30 msgr2=0x7fd9600ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9600b5d30 0x7fd9600ae7e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd960074530 tx=0x7fd95802c810 comp rx=0 tx=0).stop 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 -- 172.21.15.150:0/4211354740 shutdown_connections 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd9600b5d30 0x7fd9600ae7e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 --2- 172.21.15.150:0/4211354740 >> v2:172.21.15.187:3300/0 conn(0x7fd96006f780 0x7fd9600b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 --2- 172.21.15.150:0/4211354740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96015df00 0x7fd96006f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 -- 172.21.15.150:0/4211354740 >> 172.21.15.150:0/4211354740 conn(0x7fd9600d1ae0 msgr2=0x7fd9600d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 -- 172.21.15.150:0/4211354740 shutdown_connections 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.086+0000 7fd966134640 1 -- 172.21.15.150:0/4211354740 wait complete. 2024-04-25T14:02:57.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.088+0000 7fd966134640 1 Processor -- start 2024-04-25T14:02:57.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.088+0000 7fd966134640 1 -- start start 2024-04-25T14:02:57.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96006f780 0x7fd9601f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 0x7fd960156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd9600b4ff0 con 0x7fd96015df00 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd9600b4d10 con 0x7fd9600b5d30 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd966134640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd9600aed20 con 0x7fd96006f780 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:44988/0 (socket says 172.21.15.150:44988) 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 -- 172.21.15.150:0/2422397031 learned_addr learned my addr 172.21.15.150:0/2422397031 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95effd640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 0x7fd960156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95f7fe640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96006f780 0x7fd9601f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 -- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96006f780 msgr2=0x7fd9601f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96006f780 0x7fd9601f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 -- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 msgr2=0x7fd960156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 0x7fd960156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95ffff640 1 -- 172.21.15.150:0/2422397031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9601f7070 con 0x7fd96015df00 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.089+0000 7fd95effd640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 0x7fd960156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd95ffff640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd95800d240 tx=0x7fd958004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958044070 con 0x7fd96015df00 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd958004040 con 0x7fd96015df00 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9600cd470 con 0x7fd96015df00 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958039070 con 0x7fd96015df00 2024-04-25T14:02:57.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.090+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9600cdea0 con 0x7fd96015df00 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.091+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd960076a40 con 0x7fd96015df00 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.092+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd958047070 con 0x7fd96015df00 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.092+0000 7fd947fff640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 0x7fd94c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.092+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd9580b5bc0 con 0x7fd96015df00 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.092+0000 7fd95f7fe640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 0x7fd94c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.093+0000 7fd95f7fe640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 0x7fd94c0769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.097+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9601f7070 con 0x7fd96015df00 2024-04-25T14:02:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.276+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9600cd200 con 0x7fd96015df00 2024-04-25T14:02:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.277+0000 7fd947fff640 1 -- 172.21.15.150:0/2422397031 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fd958084e20 con 0x7fd96015df00 2024-04-25T14:02:57.277 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:02:57.277 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:02:57.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.279+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 msgr2=0x7fd94c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.279+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 0x7fd94c0769c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd950006fd0 tx=0x7fd950008040 comp rx=0 tx=0).stop 2024-04-25T14:02:57.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.279+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 msgr2=0x7fd9600d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd95800d240 tx=0x7fd958004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 shutdown_connections 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd94c0744b0 0x7fd94c0769c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd96015df00 0x7fd9600d0050 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd9600b5d30 0x7fd960156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 --2- 172.21.15.150:0/2422397031 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd96006f780 0x7fd9601f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 >> 172.21.15.150:0/2422397031 conn(0x7fd9600d1ae0 msgr2=0x7fd9600739e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 shutdown_connections 2024-04-25T14:02:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.280+0000 7fd966134640 1 -- 172.21.15.150:0/2422397031 wait complete. 2024-04-25T14:02:57.291 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:02:57.291 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:02:57.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.403+0000 7fb3732e0640 1 Processor -- start 2024-04-25T14:02:57.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.403+0000 7fb3732e0640 1 -- start start 2024-04-25T14:02:57.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb3732e0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb36c0b5ad0 0x7fb36c06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb3732e0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c06ff50 con 0x7fb36c163d80 2024-04-25T14:02:57.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb3732e0640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36c06f3d0 con 0x7fb36c160460 2024-04-25T14:02:57.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb3732e0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb36c06f6b0 con 0x7fb36c0b5ad0 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37087b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb36c0b5ad0 0x7fb36c06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37187d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb36c163d80 0x7fb36c0d1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:55254/0 (socket says 172.21.15.150:55254) 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37187d640 1 -- 172.21.15.150:0/2677164836 learned_addr learned my addr 172.21.15.150:0/2677164836 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37087b640 1 -- 172.21.15.150:0/2677164836 >> v1:172.21.15.187:6789/0 conn(0x7fb36c160460 legacy=0x7fb36c0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37087b640 1 -- 172.21.15.150:0/2677164836 >> v1:172.21.15.150:6789/0 conn(0x7fb36c163d80 legacy=0x7fb36c0d1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb363fff640 1 -- 172.21.15.150:0/2677164836 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1674248183 0 0) 0x7fb36c06ff50 con 0x7fb36c163d80 2024-04-25T14:02:57.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.405+0000 7fb37087b640 1 -- 172.21.15.150:0/2677164836 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c06f990 con 0x7fb36c0b5ad0 2024-04-25T14:02:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.406+0000 7fb37087b640 1 --2- 172.21.15.150:0/2677164836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b5ad0 0x7fb36c06ed40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0097a0 tx=0x7fb35c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5bfdeef18c8707 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.406+0000 7fb363fff640 1 -- 172.21.15.150:0/2677164836 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c037070 con 0x7fb36c0b5ad0 2024-04-25T14:02:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.406+0000 7fb363fff640 1 -- 172.21.15.150:0/2677164836 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c032440 con 0x7fb36c0b5ad0 2024-04-25T14:02:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.406+0000 7fb363fff640 1 -- 172.21.15.150:0/2677164836 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c032810 con 0x7fb36c0b5ad0 2024-04-25T14:02:57.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 -- 172.21.15.150:0/2677164836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b5ad0 msgr2=0x7fb36c06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 --2- 172.21.15.150:0/2677164836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b5ad0 0x7fb36c06ed40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb35c0097a0 tx=0x7fb35c02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 -- 172.21.15.150:0/2677164836 shutdown_connections 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 --2- 172.21.15.150:0/2677164836 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b5ad0 0x7fb36c06ed40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 -- 172.21.15.150:0/2677164836 >> 172.21.15.150:0/2677164836 conn(0x7fb36c0d03c0 msgr2=0x7fb36c0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 -- 172.21.15.150:0/2677164836 shutdown_connections 2024-04-25T14:02:57.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.407+0000 7fb3732e0640 1 -- 172.21.15.150:0/2677164836 wait complete. 2024-04-25T14:02:57.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.409+0000 7fb3732e0640 1 Processor -- start 2024-04-25T14:02:57.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.409+0000 7fb3732e0640 1 -- start start 2024-04-25T14:02:57.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb36c163d80 0x7fb36c0b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb36c150d20 0x7fb36c0b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c0ce5c0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb36c0b3030 con 0x7fb36c150d20 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb3732e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb36c0d1f70 con 0x7fb36c163d80 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb37187d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb37187d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45018/0 (socket says 172.21.15.150:45018) 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb37187d640 1 -- 172.21.15.150:0/1973408771 learned_addr learned my addr 172.21.15.150:0/1973408771 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb37087b640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb36c150d20 0x7fb36c0b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.410+0000 7fb37107c640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb36c163d80 0x7fb36c0b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 -- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb36c163d80 msgr2=0x7fb36c0b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb36c163d80 0x7fb36c0b8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 -- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb36c150d20 msgr2=0x7fb36c0b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb36c150d20 0x7fb36c0b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 -- 172.21.15.150:0/1973408771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36c0b75f0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb37187d640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c410 tx=0x7fb36400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3640074e0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3640047e0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36c0b78d0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb364004bd0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.411+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36c0b7e90 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.412+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36c0b7360 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.413+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb364011070 con 0x7fb36c0b07d0 2024-04-25T14:02:57.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.413+0000 7fb361ffb640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 0x7fb3500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.413+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fb364094f80 con 0x7fb36c0b07d0 2024-04-25T14:02:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.413+0000 7fb37107c640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 0x7fb3500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.414+0000 7fb37107c640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 0x7fb3500769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3680012e0 tx=0x7fb368009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.418+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb36405ed50 con 0x7fb36c0b07d0 2024-04-25T14:02:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.595+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb36c0b98c0 con 0x7fb36c0b07d0 2024-04-25T14:02:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.596+0000 7fb361ffb640 1 -- 172.21.15.150:0/1973408771 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fb364063050 con 0x7fb36c0b07d0 2024-04-25T14:02:57.596 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 msgr2=0x7fb3500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 0x7fb3500769c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb3680012e0 tx=0x7fb368009040 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 msgr2=0x7fb36c1609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb36400c410 tx=0x7fb36400c8d0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 shutdown_connections 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3500744b0 0x7fb3500769c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb36c0b07d0 0x7fb36c1609c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb36c150d20 0x7fb36c0b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 --2- 172.21.15.150:0/1973408771 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb36c163d80 0x7fb36c0b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 >> 172.21.15.150:0/1973408771 conn(0x7fb36c0d03c0 msgr2=0x7fb36c0d1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 shutdown_connections 2024-04-25T14:02:57.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.598+0000 7fb3732e0640 1 -- 172.21.15.150:0/1973408771 wait complete. 2024-04-25T14:02:57.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:02:57.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.722+0000 7f776e9d7640 1 Processor -- start 2024-04-25T14:02:57.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.722+0000 7f776e9d7640 1 -- start start 2024-04-25T14:02:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776e9d7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77680735b0 0x7f77681603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776e9d7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77680b8300 0x7f77680b8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776e9d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7768164a50 con 0x7f77680b6800 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776e9d7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f77680d2e90 con 0x7f77680b8300 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776e9d7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f77680d0080 con 0x7f77680735b0 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f776cf74640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f77680b8300 0x7f77680b8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f7767fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f77680b6800 0x7f7768072ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34918/0 (socket says 172.21.15.150:34918) 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f7767fff640 1 -- 172.21.15.150:0/3089351133 learned_addr learned my addr 172.21.15.150:0/3089351133 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.723+0000 7f77677fe640 1 --2- 172.21.15.150:0/3089351133 >> v2:172.21.15.187:3300/0 conn(0x7f77680735b0 0x7f77681603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f776cf74640 1 -- 172.21.15.150:0/3089351133 >> v2:172.21.15.187:3300/0 conn(0x7f77680735b0 msgr2=0x7f77681603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f776cf74640 1 --2- 172.21.15.150:0/3089351133 >> v2:172.21.15.187:3300/0 conn(0x7f77680735b0 0x7f77681603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f776cf74640 1 -- 172.21.15.150:0/3089351133 >> v1:172.21.15.150:6789/0 conn(0x7f77680b6800 legacy=0x7f7768072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f776cf74640 1 -- 172.21.15.150:0/3089351133 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7768164d30 con 0x7f77680b8300 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f7766ffd640 1 -- 172.21.15.150:0/3089351133 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1713185246 0 0) 0x7f7768164a50 con 0x7f77680b6800 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f776cf74640 1 --2- 172.21.15.150:0/3089351133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f77680b8710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7768165010 tx=0x7f776002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddb2c52690ac047 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f7766ffd640 1 -- 172.21.15.150:0/3089351133 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760007740 con 0x7f77680b8300 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.724+0000 7f7766ffd640 1 -- 172.21.15.150:0/3089351133 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77600078e0 con 0x7f77680b8300 2024-04-25T14:02:57.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f7766ffd640 1 -- 172.21.15.150:0/3089351133 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7760007cb0 con 0x7f77680b8300 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 -- 172.21.15.150:0/3089351133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 msgr2=0x7f77680b8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 --2- 172.21.15.150:0/3089351133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f77680b8710 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7768165010 tx=0x7f776002a5f0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 -- 172.21.15.150:0/3089351133 shutdown_connections 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 --2- 172.21.15.150:0/3089351133 >> v2:172.21.15.187:3300/0 conn(0x7f77680735b0 0x7f77681603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 --2- 172.21.15.150:0/3089351133 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f77680b8710 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 -- 172.21.15.150:0/3089351133 >> 172.21.15.150:0/3089351133 conn(0x7f77680d39b0 msgr2=0x7f77680d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 -- 172.21.15.150:0/3089351133 shutdown_connections 2024-04-25T14:02:57.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.725+0000 7f776e9d7640 1 -- 172.21.15.150:0/3089351133 wait complete. 2024-04-25T14:02:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.727+0000 7f776e9d7640 1 Processor -- start 2024-04-25T14:02:57.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.727+0000 7f776e9d7640 1 -- start start 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b6800 0x7f776815cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77680b1460 0x7f77680b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f77680b98e0 con 0x7f77680b8300 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f77680cfd10 con 0x7f77680b1460 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.728+0000 7f776e9d7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f77680d3230 con 0x7f77680b6800 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40410/0 (socket says 172.21.15.150:40410) 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 -- 172.21.15.150:0/4154991516 learned_addr learned my addr 172.21.15.150:0/4154991516 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f7767fff640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b6800 0x7f776815cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f776cf74640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77680b1460 0x7f77680b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 -- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b6800 msgr2=0x7f776815cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b6800 0x7f776815cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 -- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77680b1460 msgr2=0x7f77680b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77680b1460 0x7f77680b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 -- 172.21.15.150:0/4154991516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77680b03c0 con 0x7f77680b8300 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f77677fe640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f775c00e8a0 tx=0x7f775c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c017070 con 0x7f77680b8300 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775c00ee90 con 0x7f77680b8300 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77680b06a0 con 0x7f77680b8300 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f775c00cb60 con 0x7f77680b8300 2024-04-25T14:02:57.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.729+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77680b0c60 con 0x7f77680b8300 2024-04-25T14:02:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.730+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7768076a40 con 0x7f77680b8300 2024-04-25T14:02:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.731+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f775c01a070 con 0x7f77680b8300 2024-04-25T14:02:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.731+0000 7f7764ff9640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:57.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.731+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f775c095720 con 0x7f77680b8300 2024-04-25T14:02:57.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.731+0000 7f7767fff640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:57.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.732+0000 7f7767fff640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7758007fd0 tx=0x7f7758007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:57.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.736+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f775c05f460 con 0x7f77680b8300 2024-04-25T14:02:57.908 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.909+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77680b0150 con 0x7f77680b8300 2024-04-25T14:02:57.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.910+0000 7f7764ff9640 1 -- 172.21.15.150:0/4154991516 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f775c063760 con 0x7f77680b8300 2024-04-25T14:02:57.909 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 msgr2=0x7f77540769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7758007fd0 tx=0x7f7758007de0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 msgr2=0x7f776814e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f775c00e8a0 tx=0x7f775c00ed60 comp rx=0 tx=0).stop 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 shutdown_connections 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f77540744e0 0x7f77540769f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.911 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77680b1460 0x7f77680b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77680b8300 0x7f776814e090 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.911+0000 7f776e9d7640 1 --2- 172.21.15.150:0/4154991516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f77680b6800 0x7f776815cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:57.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.912+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 >> 172.21.15.150:0/4154991516 conn(0x7f77680d39b0 msgr2=0x7f77680b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:57.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.912+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 shutdown_connections 2024-04-25T14:02:57.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:57.912+0000 7f776e9d7640 1 -- 172.21.15.150:0/4154991516 wait complete. 2024-04-25T14:02:57.922 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:02:58.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.033+0000 7feaffa67640 1 Processor -- start 2024-04-25T14:02:58.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.033+0000 7feaffa67640 1 -- start start 2024-04-25T14:02:58.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feaffa67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf815abd0 0x7feaf80cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feaffa67640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feaf8160a70 0x7feaf8160e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feaffa67640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaf8070020 con 0x7feaf81613c0 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feaffa67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7feaf80d0fe0 con 0x7feaf815abd0 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feaffa67640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7feaf80cbd80 con 0x7feaf8160a70 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd803640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaf815abd0 0x7feaf80cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafe004640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feaf81613c0 0x7feaf815d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34920/0 (socket says 172.21.15.150:34920) 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafe004640 1 -- 172.21.15.150:0/4277065018 learned_addr learned my addr 172.21.15.150:0/4277065018 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd002640 1 --2- 172.21.15.150:0/4277065018 >> v2:172.21.15.187:3300/0 conn(0x7feaf8160a70 0x7feaf8160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd803640 1 -- 172.21.15.150:0/4277065018 >> v2:172.21.15.187:3300/0 conn(0x7feaf8160a70 msgr2=0x7feaf8160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd803640 1 --2- 172.21.15.150:0/4277065018 >> v2:172.21.15.187:3300/0 conn(0x7feaf8160a70 0x7feaf8160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd803640 1 -- 172.21.15.150:0/4277065018 >> v1:172.21.15.150:6789/0 conn(0x7feaf81613c0 legacy=0x7feaf815d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafd803640 1 -- 172.21.15.150:0/4277065018 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf80cde00 con 0x7feaf815abd0 2024-04-25T14:02:58.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.035+0000 7feafc801640 1 -- 172.21.15.150:0/4277065018 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 990585767 0 0) 0x7feaf0001110 con 0x7feaf81613c0 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.036+0000 7feafd803640 1 --2- 172.21.15.150:0/4277065018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf815abd0 0x7feaf80cd8c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feaf80b0d50 tx=0x7feaf402c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c12e8a66abcc9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.036+0000 7feafc801640 1 -- 172.21.15.150:0/4277065018 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4009540 con 0x7feaf815abd0 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.036+0000 7feafc801640 1 -- 172.21.15.150:0/4277065018 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf40096e0 con 0x7feaf815abd0 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.036+0000 7feafc801640 1 -- 172.21.15.150:0/4277065018 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf4009ab0 con 0x7feaf815abd0 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 -- 172.21.15.150:0/4277065018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf815abd0 msgr2=0x7feaf80cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 --2- 172.21.15.150:0/4277065018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf815abd0 0x7feaf80cd8c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7feaf80b0d50 tx=0x7feaf402c7f0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 -- 172.21.15.150:0/4277065018 shutdown_connections 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 --2- 172.21.15.150:0/4277065018 >> v2:172.21.15.187:3300/0 conn(0x7feaf8160a70 0x7feaf8160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 --2- 172.21.15.150:0/4277065018 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf815abd0 0x7feaf80cd8c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 -- 172.21.15.150:0/4277065018 >> 172.21.15.150:0/4277065018 conn(0x7feaf80ce390 msgr2=0x7feaf80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 -- 172.21.15.150:0/4277065018 shutdown_connections 2024-04-25T14:02:58.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.037+0000 7feaffa67640 1 -- 172.21.15.150:0/4277065018 wait complete. 2024-04-25T14:02:58.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.039+0000 7feaffa67640 1 Processor -- start 2024-04-25T14:02:58.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.039+0000 7feaffa67640 1 -- start start 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaf815abd0 0x7feaf8155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaf8160a70 0x7feaf80b6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feaf80cfd10 con 0x7feaf81613c0 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feaf80cc2b0 con 0x7feaf815abd0 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feaffa67640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feaf80cfff0 con 0x7feaf8160a70 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.040+0000 7feafe004640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40426/0 (socket says 172.21.15.150:40426) 2024-04-25T14:02:58.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 -- 172.21.15.150:0/4038517517 learned_addr learned my addr 172.21.15.150:0/4038517517 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafd803640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaf815abd0 0x7feaf8155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafd002640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaf8160a70 0x7feaf80b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 -- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaf8160a70 msgr2=0x7feaf80b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaf8160a70 0x7feaf80b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 -- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaf815abd0 msgr2=0x7feaf8155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaf815abd0 0x7feaf8155260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 -- 172.21.15.150:0/4038517517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaf80af5b0 con 0x7feaf81613c0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feafe004640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feaf000c3e0 tx=0x7feaf000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf00074e0 con 0x7feaf81613c0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaf00047e0 con 0x7feaf81613c0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaf80af890 con 0x7feaf81613c0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0004bd0 con 0x7feaf81613c0 2024-04-25T14:02:58.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.041+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaf80afe50 con 0x7feaf81613c0 2024-04-25T14:02:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.042+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaf8076a40 con 0x7feaf81613c0 2024-04-25T14:02:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.043+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feaf0011070 con 0x7feaf81613c0 2024-04-25T14:02:58.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.043+0000 7feaee7fc640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 0x7feadc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.043+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7feaf0094f70 con 0x7feaf81613c0 2024-04-25T14:02:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.043+0000 7feafd803640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 0x7feadc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.044+0000 7feafd803640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 0x7feadc0769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feaf4036000 tx=0x7feaf400b000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.048+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feaf8070020 con 0x7feaf81613c0 2024-04-25T14:02:58.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.225+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7feaf809ca50 con 0x7feaf81613c0 2024-04-25T14:02:58.224 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.225+0000 7feaee7fc640 1 -- 172.21.15.150:0/4038517517 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7feaf0067410 con 0x7feaf81613c0 2024-04-25T14:02:58.224 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 msgr2=0x7feadc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 0x7feadc0769c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7feaf4036000 tx=0x7feaf400b000 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 msgr2=0x7feaf80b6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7feaf000c3e0 tx=0x7feaf000c8a0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 shutdown_connections 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7feadc0744b0 0x7feadc0769c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaf81613c0 0x7feaf80b6a50 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaf8160a70 0x7feaf80b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 --2- 172.21.15.150:0/4038517517 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaf815abd0 0x7feaf8155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.227 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 >> 172.21.15.150:0/4038517517 conn(0x7feaf80ce390 msgr2=0x7feaf8070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:58.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 shutdown_connections 2024-04-25T14:02:58.228 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.227+0000 7feaffa67640 1 -- 172.21.15.150:0/4038517517 wait complete. 2024-04-25T14:02:58.238 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:02:58.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.351+0000 7f52ee4a7640 1 Processor -- start 2024-04-25T14:02:58.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.351+0000 7f52ee4a7640 1 -- start start 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52ee4a7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52e80cf3d0 0x7f52e8070120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52ee4a7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e815e360 0x7f52e8070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52ee4a7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52e80d15d0 con 0x7f52e8161d80 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52ee4a7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f52e80ce440 con 0x7f52e80cf3d0 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52ee4a7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f52e80d1870 con 0x7f52e815e360 2024-04-25T14:02:58.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52e7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52e80cf3d0 0x7f52e8070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52e7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f52e80cf3d0 0x7f52e8070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40432/0 (socket says 172.21.15.150:40432) 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.352+0000 7f52e7fff640 1 -- 172.21.15.150:0/2855385068 learned_addr learned my addr 172.21.15.150:0/2855385068 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e77fe640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e815e360 0x7f52e8070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e7fff640 1 -- 172.21.15.150:0/2855385068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e815e360 msgr2=0x7f52e8070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e7fff640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e815e360 0x7f52e8070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e7fff640 1 -- 172.21.15.150:0/2855385068 >> v1:172.21.15.187:6789/0 conn(0x7f52e8161d80 legacy=0x7f52e8162190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e7fff640 1 -- 172.21.15.150:0/2855385068 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e80d1b50 con 0x7f52e80cf3d0 2024-04-25T14:02:58.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.353+0000 7f52e7fff640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e80cf3d0 0x7f52e8070120 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009870 tx=0x7f52d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df59f5519899212 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52e6ffd640 1 -- 172.21.15.150:0/2855385068 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8037070 con 0x7f52e80cf3d0 2024-04-25T14:02:58.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52e6ffd640 1 -- 172.21.15.150:0/2855385068 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52d8032440 con 0x7f52e80cf3d0 2024-04-25T14:02:58.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52e6ffd640 1 -- 172.21.15.150:0/2855385068 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52d8032810 con 0x7f52e80cf3d0 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 -- 172.21.15.150:0/2855385068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e80cf3d0 msgr2=0x7f52e8070120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e80cf3d0 0x7f52e8070120 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009870 tx=0x7f52d802a270 comp rx=0 tx=0).stop 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 -- 172.21.15.150:0/2855385068 shutdown_connections 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e815e360 0x7f52e8070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/2855385068 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e80cf3d0 0x7f52e8070120 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 -- 172.21.15.150:0/2855385068 >> 172.21.15.150:0/2855385068 conn(0x7f52e80d3960 msgr2=0x7f52e80d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 -- 172.21.15.150:0/2855385068 shutdown_connections 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.354+0000 7f52ee4a7640 1 -- 172.21.15.150:0/2855385068 wait complete. 2024-04-25T14:02:58.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.356+0000 7f52ee4a7640 1 Processor -- start 2024-04-25T14:02:58.356 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.356+0000 7f52ee4a7640 1 -- start start 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.357+0000 7f52ee4a7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 0x7f52e815f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.357+0000 7f52ee4a7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.357+0000 7f52ee4a7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e8161d80 0x7f52e81538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.357+0000 7f52ee4a7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8070eb0 con 0x7f52e815e360 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52ee4a7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f52e8070d30 con 0x7f52e80cf3d0 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52ee4a7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f52e80cdfb0 con 0x7f52e8161d80 2024-04-25T14:02:58.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40434/0 (socket says 172.21.15.150:40434) 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 -- 172.21.15.150:0/1215216648 learned_addr learned my addr 172.21.15.150:0/1215216648 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e7fff640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 0x7f52e815f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52eca44640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e8161d80 0x7f52e81538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 -- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e8161d80 msgr2=0x7f52e81538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e8161d80 0x7f52e81538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 -- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 msgr2=0x7f52e815f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 0x7f52e815f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 -- 172.21.15.150:0/1215216648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52e81f7f70 con 0x7f52e815e360 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e7fff640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 0x7f52e815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e77fe640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00e8a0 tx=0x7f52dc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc017070 con 0x7f52e815e360 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52dc00c940 con 0x7f52e815e360 2024-04-25T14:02:58.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52e81f8250 con 0x7f52e815e360 2024-04-25T14:02:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52dc00cd30 con 0x7f52e815e360 2024-04-25T14:02:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.358+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52e81f8810 con 0x7f52e815e360 2024-04-25T14:02:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.359+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52e815f710 con 0x7f52e815e360 2024-04-25T14:02:58.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.360+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f52dc01a070 con 0x7f52e815e360 2024-04-25T14:02:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.360+0000 7f52e4ff9640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 0x7f52d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.361+0000 7f52e7fff640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 0x7f52d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.361+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f52dc095810 con 0x7f52e815e360 2024-04-25T14:02:58.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.361+0000 7f52e7fff640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 0x7f52d40769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52d8034000 tx=0x7f52d802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.365+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52dc05fc30 con 0x7f52e815e360 2024-04-25T14:02:58.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.541+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f52e8076a40 con 0x7f52e815e360 2024-04-25T14:02:58.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.542+0000 7f52e4ff9640 1 -- 172.21.15.150:0/1215216648 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f52dc05f530 con 0x7f52e815e360 2024-04-25T14:02:58.541 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:02:58.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 msgr2=0x7f52d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 0x7f52d40769c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52d8034000 tx=0x7f52d802e000 comp rx=0 tx=0).stop 2024-04-25T14:02:58.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 msgr2=0x7f52e8160e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f52dc00e8a0 tx=0x7f52dc00ed60 comp rx=0 tx=0).stop 2024-04-25T14:02:58.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 shutdown_connections 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f52d40744b0 0x7f52d40769c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f52e8161d80 0x7f52e81538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f52e815e360 0x7f52e8160e60 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 --2- 172.21.15.150:0/1215216648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f52e80cf3d0 0x7f52e815f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 >> 172.21.15.150:0/1215216648 conn(0x7f52e80d3960 msgr2=0x7f52e80d27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 shutdown_connections 2024-04-25T14:02:58.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.544+0000 7f52ee4a7640 1 -- 172.21.15.150:0/1215216648 wait complete. 2024-04-25T14:02:58.554 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj' 2024-04-25T14:02:58.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.663+0000 7fad01029640 1 Processor -- start 2024-04-25T14:02:58.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.663+0000 7fad01029640 1 -- start start 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7fad01029640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7facfc15df00 0x7facfc06f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7fad01029640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 0x7facfc0b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7fad01029640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facfc0b6140 con 0x7facfc0b5d30 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7fad01029640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7facfc15e310 con 0x7facfc15df00 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7fad01029640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facfc0b5360 con 0x7facfc06f780 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7facfc15df00 0x7facfc06f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7facfc15df00 0x7facfc06f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40440/0 (socket says 172.21.15.150:40440) 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 -- 172.21.15.150:0/3230067572 learned_addr learned my addr 172.21.15.150:0/3230067572 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:58.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfa59c640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 0x7facfc0b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 -- 172.21.15.150:0/3230067572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 msgr2=0x7facfc0b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 0x7facfc0b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 -- 172.21.15.150:0/3230067572 >> v1:172.21.15.187:6789/0 conn(0x7facfc0b5d30 legacy=0x7facfc0ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfad9d640 1 -- 172.21.15.150:0/3230067572 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facfc0b4750 con 0x7facfc15df00 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.665+0000 7facfa59c640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 0x7facfc0b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7facfad9d640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc15df00 0x7facfc06f240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7facec009e50 tx=0x7facec02c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a58f19a8192270 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7facf9d9b640 1 -- 172.21.15.150:0/3230067572 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec039070 con 0x7facfc15df00 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7facf9d9b640 1 -- 172.21.15.150:0/3230067572 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec034680 con 0x7facfc15df00 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7facf9d9b640 1 -- 172.21.15.150:0/3230067572 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec034a50 con 0x7facfc15df00 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7fad01029640 1 -- 172.21.15.150:0/3230067572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc15df00 msgr2=0x7facfc06f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.666+0000 7fad01029640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc15df00 0x7facfc06f240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7facec009e50 tx=0x7facec02c890 comp rx=0 tx=0).stop 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 -- 172.21.15.150:0/3230067572 shutdown_connections 2024-04-25T14:02:58.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc06f780 0x7facfc0b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 --2- 172.21.15.150:0/3230067572 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc15df00 0x7facfc06f240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 -- 172.21.15.150:0/3230067572 >> 172.21.15.150:0/3230067572 conn(0x7facfc0d1ae0 msgr2=0x7facfc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 -- 172.21.15.150:0/3230067572 shutdown_connections 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.667+0000 7fad01029640 1 -- 172.21.15.150:0/3230067572 wait complete. 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.669+0000 7fad01029640 1 Processor -- start 2024-04-25T14:02:58.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.669+0000 7fad01029640 1 -- start start 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facfc0b5d30 0x7facfc1f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 0x7facfc1f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7facfc0b4ff0 con 0x7facfc06f780 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7facfc0b4d10 con 0x7facfc0b5d30 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7fad01029640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7facfc0aeef0 con 0x7facfc15df00 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40444/0 (socket says 172.21.15.150:40444) 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 -- 172.21.15.150:0/1925724760 learned_addr learned my addr 172.21.15.150:0/1925724760 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:58.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfa59c640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facfc0b5d30 0x7facfc1f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfb59e640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 0x7facfc1f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 -- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 msgr2=0x7facfc1f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 0x7facfc1f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 -- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facfc0b5d30 msgr2=0x7facfc1f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facfc0b5d30 0x7facfc1f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfad9d640 1 -- 172.21.15.150:0/1925724760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facfc155090 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.670+0000 7facfb59e640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 0x7facfc1f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7facfad9d640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7facec037040 tx=0x7facec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec039050 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facec045070 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facfc155370 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facec009510 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.671+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facfc155930 con 0x7facfc06f780 2024-04-25T14:02:58.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.672+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facfc0cda50 con 0x7facfc06f780 2024-04-25T14:02:58.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.672+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7facec03e850 con 0x7facfc06f780 2024-04-25T14:02:58.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.673+0000 7face37fe640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 0x7face80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:58.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.673+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7facec0b8570 con 0x7facfc06f780 2024-04-25T14:02:58.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.673+0000 7facfa59c640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 0x7face80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:58.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.673+0000 7facfa59c640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 0x7face80769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7facfc0ae850 tx=0x7facf0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:58.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.677+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7facfc0cda50 con 0x7facfc06f780 2024-04-25T14:02:58.919 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:58.920+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj"]} v 0) -- 0x7facfc076a40 con 0x7facfc06f780 2024-04-25T14:02:59.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.338+0000 7face37fe640 1 -- 172.21.15.150:0/1925724760 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj"]}]=0 v69) ==== 150+0+0 (secure 0 0 0) 0x7facec082900 con 0x7facfc06f780 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 msgr2=0x7face80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 0x7face80769f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7facfc0ae850 tx=0x7facf0008040 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 msgr2=0x7facfc1e5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7facec037040 tx=0x7facec004a40 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 shutdown_connections 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7face80744e0 0x7face80769f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7facfc15df00 0x7facfc1f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7facfc0b5d30 0x7facfc1f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 --2- 172.21.15.150:0/1925724760 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7facfc06f780 0x7facfc1e5840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 >> 172.21.15.150:0/1925724760 conn(0x7facfc0d1ae0 msgr2=0x7facfc15d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:59.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 shutdown_connections 2024-04-25T14:02:59.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.340+0000 7fad01029640 1 -- 172.21.15.150:0/1925724760 wait complete. 2024-04-25T14:02:59.352 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T14:02:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.466+0000 7ff83e66e640 1 Processor -- start 2024-04-25T14:02:59.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.466+0000 7ff83e66e640 1 -- start start 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83e66e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380d3280 0x7ff838160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83e66e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8380735b0 0x7ff8380b8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83e66e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff838164250 con 0x7ff8380b6800 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83e66e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff8381644f0 con 0x7ff8380735b0 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83e66e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8381647d0 con 0x7ff8380d3280 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8380735b0 0x7ff8380b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff8380735b0 0x7ff8380b8460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40448/0 (socket says 172.21.15.150:40448) 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 -- 172.21.15.150:0/3898207414 learned_addr learned my addr 172.21.15.150:0/3898207414 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff837fff640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380d3280 0x7ff838160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 -- 172.21.15.150:0/3898207414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380d3280 msgr2=0x7ff838160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:59.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380d3280 0x7ff838160360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 -- 172.21.15.150:0/3898207414 >> v1:172.21.15.187:6789/0 conn(0x7ff8380b6800 legacy=0x7ff838072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:02:59.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.467+0000 7ff83cc0b640 1 -- 172.21.15.150:0/3898207414 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff838164ab0 con 0x7ff8380735b0 2024-04-25T14:02:59.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.468+0000 7ff83cc0b640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380735b0 0x7ff8380b8460 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff8380b3260 tx=0x7ff83002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b93db2321f9c0a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:59.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.468+0000 7ff836ffd640 1 -- 172.21.15.150:0/3898207414 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff830036070 con 0x7ff8380735b0 2024-04-25T14:02:59.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.468+0000 7ff836ffd640 1 -- 172.21.15.150:0/3898207414 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff830031600 con 0x7ff8380735b0 2024-04-25T14:02:59.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.468+0000 7ff836ffd640 1 -- 172.21.15.150:0/3898207414 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8300319d0 con 0x7ff8380735b0 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 -- 172.21.15.150:0/3898207414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380735b0 msgr2=0x7ff8380b8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380735b0 0x7ff8380b8460 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff8380b3260 tx=0x7ff83002a6a0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 -- 172.21.15.150:0/3898207414 shutdown_connections 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380735b0 0x7ff8380b8460 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 --2- 172.21.15.150:0/3898207414 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380d3280 0x7ff838160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 -- 172.21.15.150:0/3898207414 >> 172.21.15.150:0/3898207414 conn(0x7ff8380d39b0 msgr2=0x7ff8380d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:59.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 -- 172.21.15.150:0/3898207414 shutdown_connections 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.469+0000 7ff83e66e640 1 -- 172.21.15.150:0/3898207414 wait complete. 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.471+0000 7ff83e66e640 1 Processor -- start 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.471+0000 7ff83e66e640 1 -- start start 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8380b6800 0x7ff8380b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380cf300 0x7ff8380cf710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff8380ce000 con 0x7ff8380d3280 2024-04-25T14:02:59.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff838163f40 con 0x7ff8380b6800 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff83e66e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff8380b89a0 con 0x7ff8380cf300 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff8377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff8377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40460/0 (socket says 172.21.15.150:40460) 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.472+0000 7ff8377fe640 1 -- 172.21.15.150:0/2133943126 learned_addr learned my addr 172.21.15.150:0/2133943126 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff837fff640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8380b6800 0x7ff8380b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff83cc0b640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380cf300 0x7ff8380cf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 -- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380cf300 msgr2=0x7ff8380cf710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380cf300 0x7ff8380cf710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 -- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8380b6800 msgr2=0x7ff8380b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8380b6800 0x7ff8380b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 -- 172.21.15.150:0/2133943126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8380cfc50 con 0x7ff8380d3280 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff8377fe640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00b420 tx=0x7ff82c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c011830 con 0x7ff8380d3280 2024-04-25T14:02:59.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff82c016070 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8380b3dd0 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff82c011c10 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.473+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8380b4390 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.474+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8380b87c0 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.475+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff82c01b460 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.475+0000 7ff834ff9640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 0x7ff8240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.475+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff82c0951a0 con 0x7ff8380d3280 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.476+0000 7ff837fff640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 0x7ff8240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:02:59.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.476+0000 7ff837fff640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 0x7ff8240769f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff828006fd0 tx=0x7ff828008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:02:59.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.480+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff8380b87c0 con 0x7ff8380d3280 2024-04-25T14:02:59.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.656+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7ff838076a40 con 0x7ff8380d3280 2024-04-25T14:02:59.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.662+0000 7ff834ff9640 1 -- 172.21.15.150:0/2133943126 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7ff82c020020 con 0x7ff8380d3280 2024-04-25T14:02:59.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 msgr2=0x7ff8240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 0x7ff8240769f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff828006fd0 tx=0x7ff828008040 comp rx=0 tx=0).stop 2024-04-25T14:02:59.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 msgr2=0x7ff83814fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff82c00b420 tx=0x7ff82c00b8e0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 shutdown_connections 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff8240744e0 0x7ff8240769f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff8380cf300 0x7ff8380cf710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff8380d3280 0x7ff83814fc50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 --2- 172.21.15.150:0/2133943126 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff8380b6800 0x7ff8380b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 >> 172.21.15.150:0/2133943126 conn(0x7ff8380d39b0 msgr2=0x7ff83815c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 shutdown_connections 2024-04-25T14:02:59.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:02:59.664+0000 7ff83e66e640 1 -- 172.21.15.150:0/2133943126 wait complete. 2024-04-25T14:02:59.674 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T14:02:59.675 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:02:59.712 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:02:59.755 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T14:02:59.755 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:59.755 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:59.771 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:59.771 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:59.826 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:02:59.826 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:02:59.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:00.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.029+0000 7f9ef23b2640 1 Processor -- start 2024-04-25T14:03:00.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.029+0000 7f9ef23b2640 1 -- start start 2024-04-25T14:03:00.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9ef23b2640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9eec0b9070 0x7f9eec06d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9ef23b2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eec06c570 con 0x7f9eec1601d0 2024-04-25T14:03:00.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9ef23b2640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9eec1605e0 con 0x7f9eec06d830 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9ef23b2640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9eec0d5980 con 0x7f9eec0b9070 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9eebfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9eec1601d0 0x7f9eec0d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34934/0 (socket says 172.21.15.150:34934) 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.030+0000 7f9eebfff640 1 -- 172.21.15.150:0/223101412 learned_addr learned my addr 172.21.15.150:0/223101412 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeb7fe640 1 --2- 172.21.15.150:0/223101412 >> v2:172.21.15.187:3300/0 conn(0x7f9eec0b9070 0x7f9eec06d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 625414740 0 0) 0x7f9eec06c570 con 0x7f9eec1601d0 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ed4003520 con 0x7f9eec1601d0 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982722519 0 0) 0x7f9ed4003520 con 0x7f9eec1601d0 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 >> v2:172.21.15.187:3300/0 conn(0x7f9eec0b9070 msgr2=0x7f9eec06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 --2- 172.21.15.150:0/223101412 >> v2:172.21.15.187:3300/0 conn(0x7f9eec0b9070 0x7f9eec06d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 >> v1:172.21.15.187:6789/0 conn(0x7f9eec06d830 legacy=0x7f9eec163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eec06c810 con 0x7f9eec1601d0 2024-04-25T14:03:00.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f9ee0001690 con 0x7f9eec1601d0 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 >> v1:172.21.15.150:6789/0 conn(0x7f9eec1601d0 legacy=0x7f9eec0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 0x7f9ed4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ed4004800 0x7f9ed4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed4005250 0x7f9ed400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec0d5980 con 0x7f9ed40039b0 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed400c1e0 con 0x7f9ed4004800 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed400c3e0 con 0x7f9ed4005250 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 0x7f9ed4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eeb7fe640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ed4004800 0x7f9ed4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9ef094f640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed4005250 0x7f9ed400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 -- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed4005250 msgr2=0x7f9ed400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed4005250 0x7f9ed400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 -- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ed4004800 msgr2=0x7f9ed4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9ed4004800 0x7f9ed4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9eebfff640 1 -- 172.21.15.150:0/223101412 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed4003520 con 0x7f9ed40039b0 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9ef094f640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed4005250 0x7f9ed400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.031+0000 7f9ef094f640 1 -- 172.21.15.150:0/223101412 reap_dead start 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.032+0000 7f9eebfff640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 0x7f9ed4003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9eec08f020 tx=0x7f9ed4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee7a43ed8030ebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.032+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0007d30 con 0x7f9ed40039b0 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.032+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee0005af0 con 0x7f9ed40039b0 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.032+0000 7f9eeaffd640 1 -- 172.21.15.150:0/223101412 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0007070 con 0x7f9ed40039b0 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 -- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 msgr2=0x7f9ed4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 0x7f9ed4003e20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9eec08f020 tx=0x7f9ed4005830 comp rx=0 tx=0).stop 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 -- 172.21.15.150:0/223101412 shutdown_connections 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/223101412 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9ed40039b0 0x7f9ed4003e20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 -- 172.21.15.150:0/223101412 >> 172.21.15.150:0/223101412 conn(0x7f9eec0d5d90 msgr2=0x7f9eec0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 -- 172.21.15.150:0/223101412 shutdown_connections 2024-04-25T14:03:00.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.033+0000 7f9ef23b2640 1 -- 172.21.15.150:0/223101412 wait complete. 2024-04-25T14:03:00.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.035+0000 7f9ef23b2640 1 Processor -- start 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.035+0000 7f9ef23b2640 1 -- start start 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed40039b0 0x7f9eec1fa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eec15b400 0x7f9eec150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec0b9480 con 0x7f9eec0c98c0 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9eec06cdd0 con 0x7f9eec15b400 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef23b2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9eec0b9600 con 0x7f9ed40039b0 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef094f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef094f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40476/0 (socket says 172.21.15.150:40476) 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9ef094f640 1 -- 172.21.15.150:0/298691454 learned_addr learned my addr 172.21.15.150:0/298691454 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:00.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9eeb7fe640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eec15b400 0x7f9eec150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.036+0000 7f9eebfff640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed40039b0 0x7f9eec1fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 -- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed40039b0 msgr2=0x7f9eec1fa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed40039b0 0x7f9eec1fa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 -- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eec15b400 msgr2=0x7f9eec150b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eec15b400 0x7f9eec150b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 -- 172.21.15.150:0/298691454 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9eec0ca210 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef094f640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0d23a0 tx=0x7f9ee400ead0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee4003b80 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee4018070 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9eec161830 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee40050d0 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.037+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9eec0cd6f0 con 0x7f9eec0c98c0 2024-04-25T14:03:00.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.038+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9eec076a40 con 0x7f9eec0c98c0 2024-04-25T14:03:00.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.039+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9ee4005270 con 0x7f9eec0c98c0 2024-04-25T14:03:00.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.039+0000 7f9ee8ff9640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 0x7f9ed80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.039+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9ee4094e80 con 0x7f9eec0c98c0 2024-04-25T14:03:00.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.039+0000 7f9eebfff640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 0x7f9ed80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.039+0000 7f9eebfff640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 0x7f9ed80769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0032000 tx=0x7f9ee002d000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.044+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9ee4022020 con 0x7f9eec0c98c0 2024-04-25T14:03:00.218 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.219+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9eec0c9660 con 0x7f9eec0c98c0 2024-04-25T14:03:00.219 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.220+0000 7f9ee8ff9640 1 -- 172.21.15.150:0/298691454 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7f9eec0ca210 con 0x7f9eec0c98c0 2024-04-25T14:03:00.220 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:00.220 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 msgr2=0x7f9ed80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 0x7f9ed80769c0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0032000 tx=0x7f9ee002d000 comp rx=0 tx=0).stop 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 msgr2=0x7f9eec0c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9eec0d23a0 tx=0x7f9ee400ead0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 shutdown_connections 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ed80744b0 0x7f9ed80769c0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9eec0c98c0 0x7f9eec0c9cd0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.222 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9eec15b400 0x7f9eec150b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.222+0000 7f9ef23b2640 1 --2- 172.21.15.150:0/298691454 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9ed40039b0 0x7f9eec1fa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.223+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 >> 172.21.15.150:0/298691454 conn(0x7f9eec0d5d90 msgr2=0x7f9eec15e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.223+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 shutdown_connections 2024-04-25T14:03:00.223 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.223+0000 7f9ef23b2640 1 -- 172.21.15.150:0/298691454 wait complete. 2024-04-25T14:03:00.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:03:00.234 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:00.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.347+0000 7f8cc9cf9640 1 Processor -- start 2024-04-25T14:03:00.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.347+0000 7f8cc9cf9640 1 -- start start 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc9cf9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc40b4aa0 0x7f8cc415d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc9cf9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc406fa20 con 0x7f8cc406f610 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc9cf9640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8cc40cee30 con 0x7f8cc40cea20 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc9cf9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc40ce590 con 0x7f8cc40b4aa0 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8cc406f610 0x7f8cc40b3b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34940/0 (socket says 172.21.15.150:34940) 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.348+0000 7f8cc37fe640 1 -- 172.21.15.150:0/850011585 learned_addr learned my addr 172.21.15.150:0/850011585 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc3fff640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc40b4aa0 0x7f8cc415d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1368053663 0 0) 0x7f8cc406fa20 con 0x7f8cc406f610 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cac003520 con 0x7f8cc406f610 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 104668592 0 0) 0x7f8cb8001110 con 0x7f8cc40cea20 2024-04-25T14:03:00.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cc406fa20 con 0x7f8cc40cea20 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 314024698 0 0) 0x7f8cac003520 con 0x7f8cc406f610 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc40b4aa0 msgr2=0x7f8cc415d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc40b4aa0 0x7f8cc415d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 >> v1:172.21.15.187:6789/0 conn(0x7f8cc40cea20 legacy=0x7f8cc40b4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc40cd980 con 0x7f8cc406f610 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f8cb4001690 con 0x7f8cc406f610 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc3fff640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc40b4aa0 0x7f8cc415d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 >> v1:172.21.15.150:6789/0 conn(0x7f8cc406f610 legacy=0x7f8cc40b3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cac003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cac004800 0x7f8cac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cac005150 0x7f8cac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40ce590 con 0x7f8cac003b70 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8cac00bf00 con 0x7f8cac005150 2024-04-25T14:03:00.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cac00c0d0 con 0x7f8cac004800 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cac003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc2ffd640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cac004800 0x7f8cac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.349+0000 7f8cc3fff640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cac005150 0x7f8cac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 -- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cac004800 msgr2=0x7f8cac004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cac004800 0x7f8cac004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 -- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cac005150 msgr2=0x7f8cac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cac005150 0x7f8cac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 -- 172.21.15.150:0/850011585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cac003520 con 0x7f8cac003b70 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cac003fc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c240 tx=0x7f8cac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8db90506a8dfa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4007cf0 con 0x7f8cac003b70 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb4005ad0 con 0x7f8cac003b70 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.350+0000 7f8cc27fc640 1 -- 172.21.15.150:0/850011585 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4007070 con 0x7f8cac003b70 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/850011585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 msgr2=0x7f8cac003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/850011585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cac003fc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c240 tx=0x7f8cac005830 comp rx=0 tx=0).stop 2024-04-25T14:03:00.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc3fff640 1 -- 172.21.15.150:0/850011585 reap_dead start 2024-04-25T14:03:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/850011585 shutdown_connections 2024-04-25T14:03:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/850011585 >> 172.21.15.150:0/850011585 conn(0x7f8cc409b4a0 msgr2=0x7f8cc409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/850011585 shutdown_connections 2024-04-25T14:03:00.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.351+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/850011585 wait complete. 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.353+0000 7f8cc9cf9640 1 Processor -- start 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.353+0000 7f8cc9cf9640 1 -- start start 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc41545b0 0x7f8cc41e7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 0x7f8cc40af670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40ce220 con 0x7f8cac003b70 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8cc40cdf40 con 0x7f8cc4150730 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc9cf9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8cc40b4aa0 con 0x7f8cc41545b0 2024-04-25T14:03:00.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.354+0000 7f8cc37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40504/0 (socket says 172.21.15.150:40504) 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 -- 172.21.15.150:0/3958359514 learned_addr learned my addr 172.21.15.150:0/3958359514 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc3fff640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 0x7f8cc40af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc2ffd640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc41545b0 0x7f8cc41e7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 -- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc41545b0 msgr2=0x7f8cc41e7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc41545b0 0x7f8cc41e7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 -- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 msgr2=0x7f8cc40af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 0x7f8cc40af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 -- 172.21.15.150:0/3958359514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cc41524b0 con 0x7f8cac003b70 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc3fff640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 0x7f8cc40af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc37fe640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c240 tx=0x7f8cb40019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4040020 con 0x7f8cac003b70 2024-04-25T14:03:00.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8cb4001490 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cc4152790 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cb4037040 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.355+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cc4152d50 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.356+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cc4076a40 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.357+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8cb4043020 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.357+0000 7f8cabfff640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 0x7f8cb00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.358+0000 7f8cc2ffd640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 0x7f8cb00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.358+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8cb40b51d0 con 0x7f8cac003b70 2024-04-25T14:03:00.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.358+0000 7f8cc2ffd640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 0x7f8cb00769c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80012b0 tx=0x7f8cb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.362+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8cb4005aa0 con 0x7f8cac003b70 2024-04-25T14:03:00.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.537+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8cc40d33b0 con 0x7f8cac003b70 2024-04-25T14:03:00.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.537+0000 7f8cabfff640 1 -- 172.21.15.150:0/3958359514 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8cb407f560 con 0x7f8cac003b70 2024-04-25T14:03:00.537 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 msgr2=0x7f8cb00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 0x7f8cb00769c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8cb80012b0 tx=0x7f8cb8009040 comp rx=0 tx=0).stop 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 msgr2=0x7f8cc41e7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400c240 tx=0x7f8cb40019e0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 shutdown_connections 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8cb00744b0 0x7f8cb00769c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8cc4150730 0x7f8cc40af670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8cc41545b0 0x7f8cc41e7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 --2- 172.21.15.150:0/3958359514 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8cac003b70 0x7f8cc41e7220 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.539+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 >> 172.21.15.150:0/3958359514 conn(0x7f8cc409b4a0 msgr2=0x7f8cc406e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.540+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 shutdown_connections 2024-04-25T14:03:00.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.540+0000 7f8cc9cf9640 1 -- 172.21.15.150:0/3958359514 wait complete. 2024-04-25T14:03:00.550 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:03:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.662+0000 7effa8b97640 1 Processor -- start 2024-04-25T14:03:00.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.662+0000 7effa8b97640 1 -- start start 2024-04-25T14:03:00.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa8b97640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effa40d3290 0x7effa4160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa8b97640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 0x7effa40ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa8b97640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effa4164e10 con 0x7effa40cf330 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa8b97640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7effa4160770 con 0x7effa40d3290 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa8b97640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7effa40b6c70 con 0x7effa40735b0 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effa40d3290 0x7effa4160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7effa40d3290 0x7effa4160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40506/0 (socket says 172.21.15.150:40506) 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7eff9bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7effa40cf330 0x7effa4072ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34950/0 (socket says 172.21.15.150:34950) 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 -- 172.21.15.150:0/573542785 learned_addr learned my addr 172.21.15.150:0/573542785 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa2d9d640 1 --2- 172.21.15.150:0/573542785 >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 0x7effa40ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 -- 172.21.15.150:0/573542785 >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 msgr2=0x7effa40ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 --2- 172.21.15.150:0/573542785 >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 0x7effa40ce000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 -- 172.21.15.150:0/573542785 >> v1:172.21.15.150:6789/0 conn(0x7effa40cf330 legacy=0x7effa4072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa259c640 1 -- 172.21.15.150:0/573542785 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa4164050 con 0x7effa40d3290 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa1d9b640 1 -- 172.21.15.150:0/573542785 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764745985 0 0) 0x7effa4164e10 con 0x7effa40cf330 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.663+0000 7effa2d9d640 1 --2- 172.21.15.150:0/573542785 >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 0x7effa40ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:00.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.664+0000 7effa259c640 1 --2- 172.21.15.150:0/573542785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40d3290 0x7effa4160230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7effa40d3980 tx=0x7eff9402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6a72e07b917683 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.664+0000 7effa1d9b640 1 -- 172.21.15.150:0/573542785 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94037070 con 0x7effa40d3290 2024-04-25T14:03:00.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.664+0000 7effa1d9b640 1 -- 172.21.15.150:0/573542785 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff94032440 con 0x7effa40d3290 2024-04-25T14:03:00.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.664+0000 7effa1d9b640 1 -- 172.21.15.150:0/573542785 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94032810 con 0x7effa40d3290 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 -- 172.21.15.150:0/573542785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40d3290 msgr2=0x7effa4160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 --2- 172.21.15.150:0/573542785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40d3290 0x7effa4160230 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7effa40d3980 tx=0x7eff9402a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 -- 172.21.15.150:0/573542785 shutdown_connections 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 --2- 172.21.15.150:0/573542785 >> v2:172.21.15.187:3300/0 conn(0x7effa40735b0 0x7effa40ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 --2- 172.21.15.150:0/573542785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40d3290 0x7effa4160230 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 -- 172.21.15.150:0/573542785 >> 172.21.15.150:0/573542785 conn(0x7effa40d39b0 msgr2=0x7effa40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 -- 172.21.15.150:0/573542785 shutdown_connections 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.665+0000 7effa8b97640 1 -- 172.21.15.150:0/573542785 wait complete. 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.667+0000 7effa8b97640 1 Processor -- start 2024-04-25T14:03:00.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.667+0000 7effa8b97640 1 -- start start 2024-04-25T14:03:00.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.669 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 0x7effa41f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 0x7effa41f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7effa41648f0 con 0x7effa40cf330 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7effa4164610 con 0x7effa40b3f80 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa8b97640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7effa40b6800 con 0x7effa40b3b70 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40522/0 (socket says 172.21.15.150:40522) 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.668+0000 7effa259c640 1 -- 172.21.15.150:0/2259397132 learned_addr learned my addr 172.21.15.150:0/2259397132 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa2d9d640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 0x7effa41f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7eff9bfff640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 0x7effa41f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 -- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 msgr2=0x7effa41f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 0x7effa41f94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 -- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 msgr2=0x7effa41f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 0x7effa41f9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 -- 172.21.15.150:0/2259397132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effa40b4500 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa2d9d640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 0x7effa41f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7eff9bfff640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 0x7effa41f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa259c640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7eff940090b0 tx=0x7eff940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94037040 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff94004230 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94003610 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effa415f2f0 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.669+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effa40b4c50 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.670+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effa4076a40 con 0x7effa40cf330 2024-04-25T14:03:00.671 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.671+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7eff940037b0 con 0x7effa40cf330 2024-04-25T14:03:00.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.671+0000 7eff9affd640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 0x7eff840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.671+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7eff940b65a0 con 0x7effa40cf330 2024-04-25T14:03:00.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.671+0000 7eff9bfff640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 0x7eff840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.672+0000 7eff9bfff640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 0x7eff840769c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effa40b3030 tx=0x7eff8c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.674 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.676+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff940809c0 con 0x7effa40cf330 2024-04-25T14:03:00.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.849+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7effa40d3000 con 0x7effa40cf330 2024-04-25T14:03:00.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.850+0000 7eff9affd640 1 -- 172.21.15.150:0/2259397132 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7eff940802c0 con 0x7effa40cf330 2024-04-25T14:03:00.849 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:00.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 msgr2=0x7eff840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 0x7eff840769c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effa40b3030 tx=0x7eff8c009040 comp rx=0 tx=0).stop 2024-04-25T14:03:00.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 msgr2=0x7effa415edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7eff940090b0 tx=0x7eff940046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 shutdown_connections 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff840744b0 0x7eff840769c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7effa40b3f80 0x7effa41f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7effa40b3b70 0x7effa41f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 --2- 172.21.15.150:0/2259397132 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7effa40cf330 0x7effa415edb0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 >> 172.21.15.150:0/2259397132 conn(0x7effa40d39b0 msgr2=0x7effa40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 shutdown_connections 2024-04-25T14:03:00.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.852+0000 7effa8b97640 1 -- 172.21.15.150:0/2259397132 wait complete. 2024-04-25T14:03:00.863 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:03:00.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.973+0000 7fd055f1f640 1 Processor -- start 2024-04-25T14:03:00.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.973+0000 7fd055f1f640 1 -- start start 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd055f1f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd0500b2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd055f1f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd05015e9c0 con 0x7fd05006f640 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd055f1f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd05015eca0 con 0x7fd05009b4a0 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd055f1f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd05015ef80 con 0x7fd05015e4e0 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04effd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd05006f640 0x7fd05006cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34952/0 (socket says 172.21.15.150:34952) 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04effd640 1 -- 172.21.15.150:0/909321916 learned_addr learned my addr 172.21.15.150:0/909321916 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd0500b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706841474 0 0) 0x7fd05015e9c0 con 0x7fd05006f640 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd038003520 con 0x7fd05006f640 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3738645750 0 0) 0x7fd05015eca0 con 0x7fd05009b4a0 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd05015e9c0 con 0x7fd05009b4a0 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1688907714 0 0) 0x7fd038003520 con 0x7fd05006f640 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 msgr2=0x7fd0500b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd0500b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 >> v1:172.21.15.187:6789/0 conn(0x7fd05009b4a0 legacy=0x7fd05009b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05015f260 con 0x7fd05006f640 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fd044001670 con 0x7fd05006f640 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd0500b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 >> v1:172.21.15.150:6789/0 conn(0x7fd05006f640 legacy=0x7fd05006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd038003b70 0x7fd038003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd038004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.975+0000 7fd04e7fc640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004d40 0x7fd038005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd05015ef80 con 0x7fd038004d40 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd03800baa0 con 0x7fd038003b70 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd03800bca0 con 0x7fd05015e4e0 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004d40 0x7fd038005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04effd640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd038004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd038003b70 0x7fd038003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 -- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 msgr2=0x7fd038004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd05015e4e0 0x7fd038004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 -- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd038003b70 msgr2=0x7fd038003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd038003b70 0x7fd038003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.976+0000 7fd04ffff640 1 -- 172.21.15.150:0/909321916 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd038003520 con 0x7fd038004d40 2024-04-25T14:03:00.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd04ffff640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004d40 0x7fd038005150 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd048001fd0 tx=0x7fd038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4a371fb9a960fcd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048001bf0 con 0x7fd038004d40 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd048003b70 con 0x7fd038004d40 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd04e7fc640 1 -- 172.21.15.150:0/909321916 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0480050d0 con 0x7fd038004d40 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 -- 172.21.15.150:0/909321916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004d40 msgr2=0x7fd038005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 --2- 172.21.15.150:0/909321916 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004d40 0x7fd038005150 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd048001fd0 tx=0x7fd038005830 comp rx=0 tx=0).stop 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd04ffff640 1 -- 172.21.15.150:0/909321916 reap_dead start 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 -- 172.21.15.150:0/909321916 shutdown_connections 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 -- 172.21.15.150:0/909321916 >> 172.21.15.150:0/909321916 conn(0x7fd0500b7680 msgr2=0x7fd0500b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 -- 172.21.15.150:0/909321916 shutdown_connections 2024-04-25T14:03:00.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.977+0000 7fd055f1f640 1 -- 172.21.15.150:0/909321916 wait complete. 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.979+0000 7fd055f1f640 1 Processor -- start 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.979+0000 7fd055f1f640 1 -- start start 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0501596c0 0x7fd0501e81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050141e50 0x7fd0501410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500b24d0 con 0x7fd050144700 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500b2350 con 0x7fd0501596c0 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd055f1f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0500b2650 con 0x7fd050141e50 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40536/0 (socket says 172.21.15.150:40536) 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 -- 172.21.15.150:0/2326143622 learned_addr learned my addr 172.21.15.150:0/2326143622 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:00.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0501596c0 0x7fd0501e81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04effd640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050141e50 0x7fd0501410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 -- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050141e50 msgr2=0x7fd0501410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050141e50 0x7fd0501410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 -- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0501596c0 msgr2=0x7fd0501e81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0501596c0 0x7fd0501e81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 -- 172.21.15.150:0/2326143622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd050145050 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd04ffff640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd048033040 tx=0x7fd048001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.981+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048001bf0 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.982+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd048003970 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.982+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd048005610 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.982+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0501e8720 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.982+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd050159da0 con 0x7fd050144700 2024-04-25T14:03:00.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.983+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd050076a40 con 0x7fd050144700 2024-04-25T14:03:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.983+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd048045020 con 0x7fd050144700 2024-04-25T14:03:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.984+0000 7fd037fff640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 0x7fd03c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.984+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fd0480b5590 con 0x7fd050144700 2024-04-25T14:03:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.984+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 0x7fd03c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:00.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.984+0000 7fd04f7fe640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 0x7fd03c0769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b8200 tx=0x7fd040009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:00.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:00.988+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd050145050 con 0x7fd050144700 2024-04-25T14:03:01.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.165+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd05006c700 con 0x7fd050144700 2024-04-25T14:03:01.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.165+0000 7fd037fff640 1 -- 172.21.15.150:0/2326143622 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fd04807f2d0 con 0x7fd050144700 2024-04-25T14:03:01.165 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 msgr2=0x7fd03c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 0x7fd03c0769c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd0500b8200 tx=0x7fd040009190 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 msgr2=0x7fd050144b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd048033040 tx=0x7fd048001650 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 shutdown_connections 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd03c0744b0 0x7fd03c0769c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd050144700 0x7fd050144b10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050141e50 0x7fd0501410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 --2- 172.21.15.150:0/2326143622 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0501596c0 0x7fd0501e81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.167+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 >> 172.21.15.150:0/2326143622 conn(0x7fd0500b7680 msgr2=0x7fd050070540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:01.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.168+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 shutdown_connections 2024-04-25T14:03:01.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.168+0000 7fd055f1f640 1 -- 172.21.15.150:0/2326143622 wait complete. 2024-04-25T14:03:01.177 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:03:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.291+0000 7f92a3cee640 1 Processor -- start 2024-04-25T14:03:01.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.291+0000 7f92a3cee640 1 -- start start 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a3cee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f929c160150 0x7f929c0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a3cee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a3cee640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f929c0d4f20 con 0x7f929c0b4c10 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a3cee640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f929c06cc60 con 0x7f929c160150 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a3cee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f929c160560 con 0x7f929c06d800 2024-04-25T14:03:01.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a1289640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f929c160150 0x7f929c0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a1289640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f929c160150 0x7f929c0b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40540/0 (socket says 172.21.15.150:40540) 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a1a8a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f929c0b4c10 0x7f929c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34956/0 (socket says 172.21.15.150:34956) 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.292+0000 7f92a1289640 1 -- 172.21.15.150:0/2783703687 learned_addr learned my addr 172.21.15.150:0/2783703687 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a228b640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a1289640 1 -- 172.21.15.150:0/2783703687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 msgr2=0x7f929c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a1289640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c0d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a1289640 1 -- 172.21.15.150:0/2783703687 >> v1:172.21.15.150:6789/0 conn(0x7f929c0b4c10 legacy=0x7f929c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a1289640 1 -- 172.21.15.150:0/2783703687 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c0d5200 con 0x7f929c160150 2024-04-25T14:03:01.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a1289640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c160150 0x7f929c0b91f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f928c009870 tx=0x7f928c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b72f19360da0599 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.293+0000 7f92a0a88640 1 -- 172.21.15.150:0/2783703687 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c037070 con 0x7f929c160150 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a0a88640 1 -- 172.21.15.150:0/2783703687 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c032440 con 0x7f929c160150 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a0a88640 1 -- 172.21.15.150:0/2783703687 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c032810 con 0x7f929c160150 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 -- 172.21.15.150:0/2783703687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c160150 msgr2=0x7f929c0b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c160150 0x7f929c0b91f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f928c009870 tx=0x7f928c02a270 comp rx=0 tx=0).stop 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 -- 172.21.15.150:0/2783703687 shutdown_connections 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 --2- 172.21.15.150:0/2783703687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c160150 0x7f929c0b91f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 -- 172.21.15.150:0/2783703687 >> 172.21.15.150:0/2783703687 conn(0x7f929c0b7690 msgr2=0x7f929c0b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:01.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 -- 172.21.15.150:0/2783703687 shutdown_connections 2024-04-25T14:03:01.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.294+0000 7f92a3cee640 1 -- 172.21.15.150:0/2783703687 wait complete. 2024-04-25T14:03:01.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.296+0000 7f92a3cee640 1 Processor -- start 2024-04-25T14:03:01.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.296+0000 7f92a3cee640 1 -- start start 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f929c160150 0x7f929c0cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f929c06c8f0 con 0x7f929c0b4c10 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f929c06c610 con 0x7f929c160150 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.297+0000 7f92a3cee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f929c06dc10 con 0x7f929c06d800 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40546/0 (socket says 172.21.15.150:40546) 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 -- 172.21.15.150:0/1621529920 learned_addr learned my addr 172.21.15.150:0/1621529920 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a228b640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f929c160150 0x7f929c0cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1a8a640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 -- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 msgr2=0x7f929c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c1438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 -- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f929c160150 msgr2=0x7f929c0cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f929c160150 0x7f929c0cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 -- 172.21.15.150:0/1621529920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c144170 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a1289640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f928c000c00 tx=0x7f928c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c041020 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c0306c0 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c144450 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c037040 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.298+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c159b10 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.299+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c076a40 con 0x7f929c0b4c10 2024-04-25T14:03:01.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.300+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f928c044020 con 0x7f929c0b4c10 2024-04-25T14:03:01.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.300+0000 7f92927fc640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 0x7f92800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.301+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f928c0b5c00 con 0x7f929c0b4c10 2024-04-25T14:03:01.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.301+0000 7f92a1a8a640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 0x7f92800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.301+0000 7f92a1a8a640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 0x7f92800769f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9298007fd0 tx=0x7f9298001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.305+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f928c0bc080 con 0x7f929c0b4c10 2024-04-25T14:03:01.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.482+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f929c0d4030 con 0x7f929c0b4c10 2024-04-25T14:03:01.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.482+0000 7f92927fc640 1 -- 172.21.15.150:0/1621529920 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f928c04b030 con 0x7f929c0b4c10 2024-04-25T14:03:01.483 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 msgr2=0x7f92800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 0x7f92800769f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9298007fd0 tx=0x7f9298001630 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 msgr2=0x7f929c15eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f928c000c00 tx=0x7f928c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 shutdown_connections 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92800744e0 0x7f92800769f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f929c160150 0x7f929c0cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f929c0b4c10 0x7f929c15eb20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 --2- 172.21.15.150:0/1621529920 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f929c06d800 0x7f929c1438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 >> 172.21.15.150:0/1621529920 conn(0x7f929c0b7690 msgr2=0x7f929c0b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 shutdown_connections 2024-04-25T14:03:01.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.484+0000 7f92a3cee640 1 -- 172.21.15.150:0/1621529920 wait complete. 2024-04-25T14:03:01.496 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:03:01.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.604+0000 7f72feaaf640 1 Processor -- start 2024-04-25T14:03:01.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.604+0000 7f72feaaf640 1 -- start start 2024-04-25T14:03:01.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80d03a0 0x7f72f80cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f8070820 0x7f72f81609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f72f80cdca0 0x7f72f8160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f72f80cff10 con 0x7f72f8070820 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f72f80cf300 con 0x7f72f80cdca0 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72feaaf640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72f80cf5e0 con 0x7f72f80d03a0 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f8070820 0x7f72f81609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f72f8070820 0x7f72f81609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40558/0 (socket says 172.21.15.150:40558) 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 -- 172.21.15.150:0/1394039607 learned_addr learned my addr 172.21.15.150:0/1394039607 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72fd04c640 1 --2- 172.21.15.150:0/1394039607 >> v2:172.21.15.187:3300/0 conn(0x7f72f80cdca0 0x7f72f8160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72fc84b640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80d03a0 0x7f72f80cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 -- 172.21.15.150:0/1394039607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80d03a0 msgr2=0x7f72f80cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80d03a0 0x7f72f80cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 -- 172.21.15.150:0/1394039607 >> v2:172.21.15.187:3300/0 conn(0x7f72f80cdca0 msgr2=0x7f72f8160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1394039607 >> v2:172.21.15.187:3300/0 conn(0x7f72f80cdca0 0x7f72f8160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.606+0000 7f72f7fff640 1 -- 172.21.15.150:0/1394039607 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f80cf8c0 con 0x7f72f8070820 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.607+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81609e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72f815afc0 tx=0x7f72ec02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b020c54dae87385 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.607+0000 7f72f77fe640 1 -- 172.21.15.150:0/1394039607 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec037070 con 0x7f72f8070820 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.607+0000 7f72f77fe640 1 -- 172.21.15.150:0/1394039607 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72ec032440 con 0x7f72f8070820 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.607+0000 7f72f77fe640 1 -- 172.21.15.150:0/1394039607 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72ec032810 con 0x7f72f8070820 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 -- 172.21.15.150:0/1394039607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 msgr2=0x7f72f81609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81609e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f72f815afc0 tx=0x7f72ec02a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 -- 172.21.15.150:0/1394039607 shutdown_connections 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1394039607 >> v2:172.21.15.187:3300/0 conn(0x7f72f80cdca0 0x7f72f8160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81609e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1394039607 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80d03a0 0x7f72f80cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 -- 172.21.15.150:0/1394039607 >> 172.21.15.150:0/1394039607 conn(0x7f72f80ce390 msgr2=0x7f72f80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:01.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 -- 172.21.15.150:0/1394039607 shutdown_connections 2024-04-25T14:03:01.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.608+0000 7f72feaaf640 1 -- 172.21.15.150:0/1394039607 wait complete. 2024-04-25T14:03:01.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.610+0000 7f72feaaf640 1 Processor -- start 2024-04-25T14:03:01.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.610+0000 7f72feaaf640 1 -- start start 2024-04-25T14:03:01.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80cdca0 0x7f72f81f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72f80d03a0 0x7f72f806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f72f815d3f0 con 0x7f72f8070820 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f72f80cc2c0 con 0x7f72f80d03a0 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72feaaf640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f72f815d570 con 0x7f72f80cdca0 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72fc84b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72fc84b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40570/0 (socket says 172.21.15.150:40570) 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72fc84b640 1 -- 172.21.15.150:0/1239105171 learned_addr learned my addr 172.21.15.150:0/1239105171 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72fd04c640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72f80d03a0 0x7f72f806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.611+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80cdca0 0x7f72f81f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 -- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80cdca0 msgr2=0x7f72f81f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80cdca0 0x7f72f81f63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 -- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72f80d03a0 msgr2=0x7f72f806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72f80d03a0 0x7f72f806ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 -- 172.21.15.150:0/1239105171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72f8155810 con 0x7f72f8070820 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72fc84b640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d600 tx=0x7f72e800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8004240 con 0x7f72f8070820 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72e80043e0 con 0x7f72f8070820 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72f81f7080 con 0x7f72f8070820 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72e8003830 con 0x7f72f8070820 2024-04-25T14:03:01.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.612+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72f81f7640 con 0x7f72f8070820 2024-04-25T14:03:01.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.613+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72f8076a40 con 0x7f72f8070820 2024-04-25T14:03:01.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.614+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f72e80039d0 con 0x7f72f8070820 2024-04-25T14:03:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.614+0000 7f72f57fa640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 0x7f72e40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.615+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 0x7f72e40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.615+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f72e8095770 con 0x7f72f8070820 2024-04-25T14:03:01.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.615+0000 7f72f7fff640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 0x7f72e40769f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72ec034000 tx=0x7f72ec02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:01.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.620+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72e805f430 con 0x7f72f8070820 2024-04-25T14:03:01.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.806+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f72f8069c60 con 0x7f72f8070820 2024-04-25T14:03:01.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.807+0000 7f72f57fa640 1 -- 172.21.15.150:0/1239105171 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f72e8063730 con 0x7f72f8070820 2024-04-25T14:03:01.807 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 msgr2=0x7f72e40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 0x7f72e40769f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f72ec034000 tx=0x7f72ec02e000 comp rx=0 tx=0).stop 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 msgr2=0x7f72f81552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f72e800d600 tx=0x7f72e800dac0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 shutdown_connections 2024-04-25T14:03:01.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f72e40744e0 0x7f72e40769f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f72f80d03a0 0x7f72f806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f72f80cdca0 0x7f72f81f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 --2- 172.21.15.150:0/1239105171 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f72f8070820 0x7f72f81552d0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 >> 172.21.15.150:0/1239105171 conn(0x7f72f80ce390 msgr2=0x7f72f815f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 shutdown_connections 2024-04-25T14:03:01.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:01.809+0000 7f72feaaf640 1 -- 172.21.15.150:0/1239105171 wait complete. 2024-04-25T14:03:01.820 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T14:03:01.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:01.821 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-04-25T14:03:01.845 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:03:01.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:01.845 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T14:03:01.900 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:01.900 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:01.919 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:01.919 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:01.977 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:01.977 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:02.036 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:02.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.142+0000 7eff7dcd1640 1 Processor -- start 2024-04-25T14:03:02.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.142+0000 7eff7dcd1640 1 -- start start 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 0x7eff7815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7eff78070e70 0x7eff7806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 0x7eff78160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7eff7806d750 con 0x7eff78070e70 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7eff780afe00 con 0x7eff78156cd0 2024-04-25T14:03:02.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff7dcd1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7eff780b00a0 con 0x7eff7806d340 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7eff78070e70 0x7eff7806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7eff78070e70 0x7eff7806ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40584/0 (socket says 172.21.15.150:40584) 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 -- 172.21.15.150:0/3664750087 learned_addr learned my addr 172.21.15.150:0/3664750087 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 -- 172.21.15.150:0/3664750087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 msgr2=0x7eff78160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff777fe640 1 --2- 172.21.15.150:0/3664750087 >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 0x7eff7815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff77fff640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 0x7eff78160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 0x7eff78160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 -- 172.21.15.150:0/3664750087 >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 msgr2=0x7eff7815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 --2- 172.21.15.150:0/3664750087 >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 0x7eff7815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff76ffd640 1 -- 172.21.15.150:0/3664750087 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff780b0380 con 0x7eff78070e70 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff777fe640 1 --2- 172.21.15.150:0/3664750087 >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 0x7eff7815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.144+0000 7eff77fff640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 0x7eff78160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:02.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.145+0000 7eff76ffd640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78070e70 0x7eff7806ce00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff78074690 tx=0x7eff6c02b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=951d2a910738e569 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.145+0000 7eff767fc640 1 -- 172.21.15.150:0/3664750087 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c00b040 con 0x7eff78070e70 2024-04-25T14:03:02.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.145+0000 7eff767fc640 1 -- 172.21.15.150:0/3664750087 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c030710 con 0x7eff78070e70 2024-04-25T14:03:02.147 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.145+0000 7eff767fc640 1 -- 172.21.15.150:0/3664750087 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c030ae0 con 0x7eff78070e70 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 -- 172.21.15.150:0/3664750087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78070e70 msgr2=0x7eff7806ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78070e70 0x7eff7806ce00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7eff78074690 tx=0x7eff6c02b5a0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 -- 172.21.15.150:0/3664750087 shutdown_connections 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff7806d340 0x7eff78160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/3664750087 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78070e70 0x7eff7806ce00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/3664750087 >> v2:172.21.15.187:3300/0 conn(0x7eff78156cd0 0x7eff7815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 -- 172.21.15.150:0/3664750087 >> 172.21.15.150:0/3664750087 conn(0x7eff780d1ae0 msgr2=0x7eff780d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 -- 172.21.15.150:0/3664750087 shutdown_connections 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.146+0000 7eff7dcd1640 1 -- 172.21.15.150:0/3664750087 wait complete. 2024-04-25T14:03:02.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.148+0000 7eff7dcd1640 1 Processor -- start 2024-04-25T14:03:02.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.148+0000 7eff7dcd1640 1 -- start start 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff7806d340 0x7eff7814b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 0x7eff7814b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7eff780b0e60 con 0x7eff78156cd0 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7eff780b0940 con 0x7eff7806d340 2024-04-25T14:03:02.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff7dcd1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7eff78071280 con 0x7eff78070e70 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff77fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40594/0 (socket says 172.21.15.150:40594) 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff77fff640 1 -- 172.21.15.150:0/2122922263 learned_addr learned my addr 172.21.15.150:0/2122922263 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff76ffd640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 0x7eff7814b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.149+0000 7eff777fe640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff7806d340 0x7eff7814b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 -- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 msgr2=0x7eff7814b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 0x7eff7814b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 -- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff7806d340 msgr2=0x7eff7814b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff7806d340 0x7eff7814b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 -- 172.21.15.150:0/2122922263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7814cad0 con 0x7eff78156cd0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff76ffd640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 0x7eff7814b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff77fff640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff7000ccd0 tx=0x7eff7000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7000f680 con 0x7eff78156cd0 2024-04-25T14:03:02.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7000f840 con 0x7eff78156cd0 2024-04-25T14:03:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff7814cdb0 con 0x7eff78156cd0 2024-04-25T14:03:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7000fc30 con 0x7eff78156cd0 2024-04-25T14:03:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.150+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff7814d370 con 0x7eff78156cd0 2024-04-25T14:03:02.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.151+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff78076a40 con 0x7eff78156cd0 2024-04-25T14:03:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.152+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7eff700213f0 con 0x7eff78156cd0 2024-04-25T14:03:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.152+0000 7eff5ffff640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 0x7eff640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.152+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7eff70096320 con 0x7eff78156cd0 2024-04-25T14:03:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.152+0000 7eff777fe640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 0x7eff640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.153+0000 7eff777fe640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 0x7eff640769c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff78074640 tx=0x7eff68008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.157+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff70014030 con 0x7eff78156cd0 2024-04-25T14:03:02.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.334+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff780b61d0 con 0x7eff78156cd0 2024-04-25T14:03:02.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.335+0000 7eff5ffff640 1 -- 172.21.15.150:0/2122922263 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7eff70060740 con 0x7eff78156cd0 2024-04-25T14:03:02.336 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:02.336 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:02.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 msgr2=0x7eff640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 0x7eff640769c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7eff78074640 tx=0x7eff68008040 comp rx=0 tx=0).stop 2024-04-25T14:03:02.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 msgr2=0x7eff780b1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7eff7000ccd0 tx=0x7eff7000a650 comp rx=0 tx=0).stop 2024-04-25T14:03:02.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 shutdown_connections 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7eff640744b0 0x7eff640769c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7eff78156cd0 0x7eff780b1190 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7eff78070e70 0x7eff7814b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 --2- 172.21.15.150:0/2122922263 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7eff7806d340 0x7eff7814b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 >> 172.21.15.150:0/2122922263 conn(0x7eff780d1ae0 msgr2=0x7eff78151ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 shutdown_connections 2024-04-25T14:03:02.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.338+0000 7eff7dcd1640 1 -- 172.21.15.150:0/2122922263 wait complete. 2024-04-25T14:03:02.350 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:03:02.350 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:02.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.461+0000 7fa476ae7640 1 Processor -- start 2024-04-25T14:03:02.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.461+0000 7fa476ae7640 1 -- start start 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa476ae7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa47015a850 0x7fa47009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa476ae7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47015ac60 con 0x7fa470160770 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa476ae7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47009bdf0 con 0x7fa47009b9e0 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa476ae7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa470160ce0 con 0x7fa47015a850 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa46ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa47015a850 0x7fa47009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa474883640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa470160770 0x7fa47006f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34970/0 (socket says 172.21.15.150:34970) 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa474883640 1 -- 172.21.15.150:0/978849297 learned_addr learned my addr 172.21.15.150:0/978849297 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa46ffff640 1 -- 172.21.15.150:0/978849297 >> v1:172.21.15.187:6789/0 conn(0x7fa47009b9e0 legacy=0x7fa4700b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa46ffff640 1 -- 172.21.15.150:0/978849297 >> v1:172.21.15.150:6789/0 conn(0x7fa470160770 legacy=0x7fa47006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.462+0000 7fa46ffff640 1 -- 172.21.15.150:0/978849297 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa470160fc0 con 0x7fa47015a850 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.463+0000 7fa46ffff640 1 --2- 172.21.15.150:0/978849297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa47015a850 0x7fa47009b4a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa4640097a0 tx=0x7fa46402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d17ee45e7d97433 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.463+0000 7fa46f7fe640 1 -- 172.21.15.150:0/978849297 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464037070 con 0x7fa47015a850 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.463+0000 7fa46f7fe640 1 -- 172.21.15.150:0/978849297 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa464032440 con 0x7fa47015a850 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.463+0000 7fa46f7fe640 1 -- 172.21.15.150:0/978849297 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464032810 con 0x7fa47015a850 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 -- 172.21.15.150:0/978849297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa47015a850 msgr2=0x7fa47009b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 --2- 172.21.15.150:0/978849297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa47015a850 0x7fa47009b4a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa4640097a0 tx=0x7fa46402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 -- 172.21.15.150:0/978849297 shutdown_connections 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 --2- 172.21.15.150:0/978849297 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa47015a850 0x7fa47009b4a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 -- 172.21.15.150:0/978849297 >> 172.21.15.150:0/978849297 conn(0x7fa4700b7680 msgr2=0x7fa4700b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 -- 172.21.15.150:0/978849297 shutdown_connections 2024-04-25T14:03:02.465 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.464+0000 7fa476ae7640 1 -- 172.21.15.150:0/978849297 wait complete. 2024-04-25T14:03:02.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.466+0000 7fa476ae7640 1 Processor -- start 2024-04-25T14:03:02.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.466+0000 7fa476ae7640 1 -- start start 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47009b9e0 0x7fa470156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa47015a850 0x7fa4701503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa4700b3780 con 0x7fa470160770 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa470161580 con 0x7fa47009b9e0 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa476ae7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4700b3900 con 0x7fa47015a850 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa475084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa475084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40604/0 (socket says 172.21.15.150:40604) 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa475084640 1 -- 172.21.15.150:0/2471387577 learned_addr learned my addr 172.21.15.150:0/2471387577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa474883640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47009b9e0 0x7fa470156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.467+0000 7fa46ffff640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa47015a850 0x7fa4701503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 -- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa47015a850 msgr2=0x7fa4701503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa47015a850 0x7fa4701503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 -- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47009b9e0 msgr2=0x7fa470156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47009b9e0 0x7fa470156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 -- 172.21.15.150:0/2471387577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa470157230 con 0x7fa470160770 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa475084640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c680 tx=0x7fa46800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468003b70 con 0x7fa470160770 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa468003d10 con 0x7fa470160770 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa470157510 con 0x7fa470160770 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa468005310 con 0x7fa470160770 2024-04-25T14:03:02.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.468+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa470157ad0 con 0x7fa470160770 2024-04-25T14:03:02.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.469+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa470076a40 con 0x7fa470160770 2024-04-25T14:03:02.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.470+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa4680054b0 con 0x7fa470160770 2024-04-25T14:03:02.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.470+0000 7fa46d7fa640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 0x7fa45c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.470+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa4680952f0 con 0x7fa470160770 2024-04-25T14:03:02.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.470+0000 7fa474883640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 0x7fa45c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.471+0000 7fa474883640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 0x7fa45c0769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa460005880 tx=0x7fa460009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.475+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa47009bdf0 con 0x7fa470160770 2024-04-25T14:03:02.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.650+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa47006f8e0 con 0x7fa470160770 2024-04-25T14:03:02.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.650+0000 7fa46d7fa640 1 -- 172.21.15.150:0/2471387577 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fa46805f030 con 0x7fa470160770 2024-04-25T14:03:02.651 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 msgr2=0x7fa45c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 0x7fa45c0769c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa460005880 tx=0x7fa460009040 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 msgr2=0x7fa4701517b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c680 tx=0x7fa46800cb40 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 shutdown_connections 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa45c0744b0 0x7fa45c0769c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa470160770 0x7fa4701517b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa47015a850 0x7fa4701503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 --2- 172.21.15.150:0/2471387577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa47009b9e0 0x7fa470156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 >> 172.21.15.150:0/2471387577 conn(0x7fa4700b7680 msgr2=0x7fa470161980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 shutdown_connections 2024-04-25T14:03:02.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.652+0000 7fa476ae7640 1 -- 172.21.15.150:0/2471387577 wait complete. 2024-04-25T14:03:02.664 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.774+0000 7f47ed934640 1 Processor -- start 2024-04-25T14:03:02.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.774+0000 7f47ed934640 1 -- start start 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.775+0000 7f47ed934640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e81600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.775+0000 7f47ed934640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 0x7f47e806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.775+0000 7f47ed934640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e80d1750 con 0x7f47e8160630 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.775+0000 7f47ed934640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f47e8160a40 con 0x7f47e80d1000 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.775+0000 7f47ed934640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47e80d55d0 con 0x7f47e80d3de0 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e77fe640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 0x7f47e806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47de7fc640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f47e8160630 0x7f47e80d08f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:34226/0 (socket says 172.21.15.150:34226) 2024-04-25T14:03:02.777 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e6ffd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e81600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47de7fc640 1 -- 172.21.15.150:0/2115809214 learned_addr learned my addr 172.21.15.150:0/2115809214 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e67fc640 1 -- 172.21.15.150:0/2115809214 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1929003469 0 0) 0x7f47e80d1750 con 0x7f47e8160630 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e67fc640 1 -- 172.21.15.150:0/2115809214 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47d4003520 con 0x7f47e8160630 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e6ffd640 1 -- 172.21.15.150:0/2115809214 >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 msgr2=0x7f47e806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e6ffd640 1 --2- 172.21.15.150:0/2115809214 >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 0x7f47e806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e6ffd640 1 -- 172.21.15.150:0/2115809214 >> v1:172.21.15.187:6789/0 conn(0x7f47e8160630 legacy=0x7f47e80d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e6ffd640 1 -- 172.21.15.150:0/2115809214 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e80d4530 con 0x7f47e80d3de0 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.776+0000 7f47e77fe640 1 --2- 172.21.15.150:0/2115809214 >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 0x7f47e806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.777+0000 7f47e6ffd640 1 --2- 172.21.15.150:0/2115809214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e81600f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f47d80097a0 tx=0x7f47d802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7bf4dfea2392472 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.777+0000 7f47e67fc640 1 -- 172.21.15.150:0/2115809214 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d8037070 con 0x7f47e80d3de0 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.777+0000 7f47e67fc640 1 -- 172.21.15.150:0/2115809214 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d8032440 con 0x7f47e80d3de0 2024-04-25T14:03:02.778 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.777+0000 7f47e67fc640 1 -- 172.21.15.150:0/2115809214 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d80327f0 con 0x7f47e80d3de0 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 -- 172.21.15.150:0/2115809214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 msgr2=0x7f47e81600f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 --2- 172.21.15.150:0/2115809214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e81600f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f47d80097a0 tx=0x7f47d802a180 comp rx=0 tx=0).stop 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 -- 172.21.15.150:0/2115809214 shutdown_connections 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 --2- 172.21.15.150:0/2115809214 >> v2:172.21.15.187:3300/0 conn(0x7f47e80d1000 0x7f47e806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 --2- 172.21.15.150:0/2115809214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e81600f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 -- 172.21.15.150:0/2115809214 >> 172.21.15.150:0/2115809214 conn(0x7f47e80d25c0 msgr2=0x7f47e80d29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 -- 172.21.15.150:0/2115809214 shutdown_connections 2024-04-25T14:03:02.780 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.778+0000 7f47ed934640 1 -- 172.21.15.150:0/2115809214 wait complete. 2024-04-25T14:03:02.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.779+0000 7f47ed934640 1 Processor -- start 2024-04-25T14:03:02.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.780+0000 7f47ed934640 1 -- start start 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e80ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47e815c020 0x7f47e80cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f47e80d4dd0 con 0x7f47e8160630 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f47e80d4af0 con 0x7f47e815c020 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47ed934640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f47e80d50b0 con 0x7f47e80d3de0 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40606/0 (socket says 172.21.15.150:40606) 2024-04-25T14:03:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 -- 172.21.15.150:0/2821918232 learned_addr learned my addr 172.21.15.150:0/2821918232 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47e77fe640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47e815c020 0x7f47e80cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47e6ffd640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e80ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 -- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 msgr2=0x7f47e80ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e80ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 -- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47e815c020 msgr2=0x7f47e80cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47e815c020 0x7f47e80cd7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.781+0000 7f47de7fc640 1 -- 172.21.15.150:0/2821918232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e80c96d0 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47de7fc640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c390 tx=0x7f47d000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d0017070 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d0005ce0 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e80c99b0 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d00072f0 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.782+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e80c9f70 con 0x7f47e8160630 2024-04-25T14:03:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.783+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47e8076a40 con 0x7f47e8160630 2024-04-25T14:03:02.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.784+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f47d001a070 con 0x7f47e8160630 2024-04-25T14:03:02.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.784+0000 7f47e4ff9640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:02.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.784+0000 7f47e6ffd640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:02.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.784+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f47d00948e0 con 0x7f47e8160630 2024-04-25T14:03:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.785+0000 7f47e6ffd640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 0x7f47c80769c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47d8034000 tx=0x7f47d802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.789+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f47d005e620 con 0x7f47e8160630 2024-04-25T14:03:02.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.963+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f47e80d1ef0 con 0x7f47e8160630 2024-04-25T14:03:02.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.963+0000 7f47e4ff9640 1 -- 172.21.15.150:0/2821918232 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f47d0062920 con 0x7f47e8160630 2024-04-25T14:03:02.963 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 msgr2=0x7f47c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 0x7f47c80769c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f47d8034000 tx=0x7f47d802e000 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 msgr2=0x7f47e815a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f47d000c390 tx=0x7f47d000c790 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 shutdown_connections 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f47c80744b0 0x7f47c80769c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f47e815c020 0x7f47e80cd7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f47e8160630 0x7f47e815a020 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 --2- 172.21.15.150:0/2821918232 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f47e80d3de0 0x7f47e80ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 >> 172.21.15.150:0/2821918232 conn(0x7f47e80d25c0 msgr2=0x7f47e806ef70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:02.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 shutdown_connections 2024-04-25T14:03:02.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:02.965+0000 7f47ed934640 1 -- 172.21.15.150:0/2821918232 wait complete. 2024-04-25T14:03:02.977 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:03:03.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.087+0000 7ff98cee7640 1 Processor -- start 2024-04-25T14:03:03.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.087+0000 7ff98cee7640 1 -- start start 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98cee7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9881603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98cee7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff988164a50 con 0x7ff9880b8300 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98cee7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff988164d30 con 0x7ff988072ea0 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98cee7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9880d2e90 con 0x7ff9880b6800 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff986d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ff9880b8300 0x7ff9880b8710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34984/0 (socket says 172.21.15.150:34984) 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff986d9d640 1 -- 172.21.15.150:0/3426438241 learned_addr learned my addr 172.21.15.150:0/3426438241 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98659c640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9881603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 132534307 0 0) 0x7ff988164a50 con 0x7ff9880b8300 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff970003520 con 0x7ff9880b8300 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4125532350 0 0) 0x7ff970003520 con 0x7ff9880b8300 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 msgr2=0x7ff9881603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9881603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 >> v1:172.21.15.187:6789/0 conn(0x7ff988072ea0 legacy=0x7ff9880732b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9880cfa30 con 0x7ff9880b8300 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7ff980001670 con 0x7ff9880b8300 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 >> v1:172.21.15.150:6789/0 conn(0x7ff9880b8300 legacy=0x7ff9880b8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9700039e0 0x7ff970003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff970004800 0x7ff970004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.089+0000 7ff98559a640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9700052d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880d2e90 con 0x7ff9700039e0 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff97000bb80 con 0x7ff970004800 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff97000bd80 con 0x7ff9880b6800 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9700039e0 0x7ff970003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff985d9b640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff970004800 0x7ff970004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff986d9d640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9700052d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 -- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 msgr2=0x7ff9700052d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9880b6800 0x7ff9700052d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 -- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff970004800 msgr2=0x7ff970004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff970004800 0x7ff970004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.090+0000 7ff98659c640 1 -- 172.21.15.150:0/3426438241 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff970003520 con 0x7ff9700039e0 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98659c640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9700039e0 0x7ff970003e50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f90 tx=0x7ff970005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7eff820f771816 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978009c10 con 0x7ff9700039e0 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff978036070 con 0x7ff9700039e0 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98559a640 1 -- 172.21.15.150:0/3426438241 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff978031810 con 0x7ff9700039e0 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98cee7640 1 -- 172.21.15.150:0/3426438241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9700039e0 msgr2=0x7ff970003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff98cee7640 1 --2- 172.21.15.150:0/3426438241 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9700039e0 0x7ff970003e50 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff978009f90 tx=0x7ff970005830 comp rx=0 tx=0).stop 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.091+0000 7ff986d9d640 1 -- 172.21.15.150:0/3426438241 reap_dead start 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.092+0000 7ff98cee7640 1 -- 172.21.15.150:0/3426438241 shutdown_connections 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.092+0000 7ff98cee7640 1 -- 172.21.15.150:0/3426438241 >> 172.21.15.150:0/3426438241 conn(0x7ff9880d39b0 msgr2=0x7ff9880d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.092+0000 7ff98cee7640 1 -- 172.21.15.150:0/3426438241 shutdown_connections 2024-04-25T14:03:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.092+0000 7ff98cee7640 1 -- 172.21.15.150:0/3426438241 wait complete. 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.093+0000 7ff98cee7640 1 Processor -- start 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.094+0000 7ff98cee7640 1 -- start start 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 0x7ff98815ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988163f90 0x7ff9881f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880b98e0 con 0x7ff9881f7360 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff9880d0080 con 0x7ff988163f90 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98cee7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9880d3230 con 0x7ff9700039e0 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40618/0 (socket says 172.21.15.150:40618) 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 -- 172.21.15.150:0/2206207304 learned_addr learned my addr 172.21.15.150:0/2206207304 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff985d9b640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988163f90 0x7ff9881f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98659c640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 0x7ff98815ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 -- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 msgr2=0x7ff98815ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 0x7ff98815ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 -- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988163f90 msgr2=0x7ff9881f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988163f90 0x7ff9881f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 -- 172.21.15.150:0/2206207304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98815ef30 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff98659c640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 0x7ff98815ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.095+0000 7ff986d9d640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff9880b3260 tx=0x7ff98000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.096+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff980019070 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.096+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff98000c690 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.096+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98815f210 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.096+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff98000ca80 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.096+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98815f7d0 con 0x7ff9881f7360 2024-04-25T14:03:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.097+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff98815d5a0 con 0x7ff9881f7360 2024-04-25T14:03:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.097+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff98001c070 con 0x7ff9881f7360 2024-04-25T14:03:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.098+0000 7ff976ffd640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 0x7ff9680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.098+0000 7ff98659c640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 0x7ff9680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.098+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7ff980094b20 con 0x7ff9881f7360 2024-04-25T14:03:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.098+0000 7ff98659c640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 0x7ff9680769f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff978035000 tx=0x7ff97802f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.103+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff98005eeb0 con 0x7ff9881f7360 2024-04-25T14:03:03.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.277+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff988070900 con 0x7ff9881f7360 2024-04-25T14:03:03.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.278+0000 7ff976ffd640 1 -- 172.21.15.150:0/2206207304 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7ff988070900 con 0x7ff9881f7360 2024-04-25T14:03:03.278 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:03:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 msgr2=0x7ff9680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 0x7ff9680769f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff978035000 tx=0x7ff97802f000 comp rx=0 tx=0).stop 2024-04-25T14:03:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 msgr2=0x7ff988153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff9880b3260 tx=0x7ff98000eb90 comp rx=0 tx=0).stop 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 shutdown_connections 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9680744e0 0x7ff9680769f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9881f7360 0x7ff988153ee0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff988163f90 0x7ff9881f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 --2- 172.21.15.150:0/2206207304 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9700039e0 0x7ff98815ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 >> 172.21.15.150:0/2206207304 conn(0x7ff9880d39b0 msgr2=0x7ff9880d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 shutdown_connections 2024-04-25T14:03:03.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.280+0000 7ff98cee7640 1 -- 172.21.15.150:0/2206207304 wait complete. 2024-04-25T14:03:03.292 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:03:03.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.403+0000 7fee8ace2640 1 Processor -- start 2024-04-25T14:03:03.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.403+0000 7fee8ace2640 1 -- start start 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee8ace2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee8409b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee8ace2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8409bdf0 con 0x7fee8415e4e0 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee8ace2640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee8415e9c0 con 0x7fee8406f640 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee8ace2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee8415eca0 con 0x7fee8409b4a0 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee88a7e640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fee8415e4e0 0x7fee840b2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34990/0 (socket says 172.21.15.150:34990) 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee88a7e640 1 -- 172.21.15.150:0/1551900208 learned_addr learned my addr 172.21.15.150:0/1551900208 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1098748223 0 0) 0x7fee74001110 con 0x7fee8415e4e0 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee6c003520 con 0x7fee8415e4e0 2024-04-25T14:03:03.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee8927f640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee8409b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3992177005 0 0) 0x7fee6c003520 con 0x7fee8415e4e0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 msgr2=0x7fee8409b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee8409b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 >> v1:172.21.15.187:6789/0 conn(0x7fee8406f640 legacy=0x7fee8406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8415ef80 con 0x7fee8415e4e0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fee74001840 con 0x7fee8415e4e0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.405+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 >> v1:172.21.15.150:6789/0 conn(0x7fee8415e4e0 legacy=0x7fee840b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6c0039e0 0x7fee6c003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee6c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c004d40 0x7fee6c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee8415eca0 con 0x7fee6c004d40 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee6c00bbf0 con 0x7fee6c0039e0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee6c00bdf0 con 0x7fee8409b4a0 2024-04-25T14:03:03.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c004d40 0x7fee6c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee88a7e640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6c0039e0 0x7fee6c003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee83fff640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee6c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 -- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 msgr2=0x7fee6c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee6c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 -- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6c0039e0 msgr2=0x7fee6c003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee6c0039e0 0x7fee6c003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee8927f640 1 -- 172.21.15.150:0/1551900208 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6c003520 con 0x7fee6c004d40 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.406+0000 7fee83fff640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee8409b4a0 0x7fee6c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8927f640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c004d40 0x7fee6c005210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fee7c007360 tx=0x7fee6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492c42224ca3d57b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c009be0 con 0x7fee6c004d40 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7c036600 con 0x7fee6c004d40 2024-04-25T14:03:03.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee837fe640 1 -- 172.21.15.150:0/1551900208 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c0369d0 con 0x7fee6c004d40 2024-04-25T14:03:03.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8ace2640 1 -- 172.21.15.150:0/1551900208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c004d40 msgr2=0x7fee6c005210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8ace2640 1 --2- 172.21.15.150:0/1551900208 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee6c004d40 0x7fee6c005210 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fee7c007360 tx=0x7fee6c005830 comp rx=0 tx=0).stop 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8927f640 1 -- 172.21.15.150:0/1551900208 reap_dead start 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8ace2640 1 -- 172.21.15.150:0/1551900208 shutdown_connections 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.407+0000 7fee8ace2640 1 -- 172.21.15.150:0/1551900208 >> 172.21.15.150:0/1551900208 conn(0x7fee840b7680 msgr2=0x7fee840b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.408+0000 7fee8ace2640 1 -- 172.21.15.150:0/1551900208 shutdown_connections 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.408+0000 7fee8ace2640 1 -- 172.21.15.150:0/1551900208 wait complete. 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.409+0000 7fee8ace2640 1 Processor -- start 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.409+0000 7fee8ace2640 1 -- start start 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee840b2610 0x7fee840b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee840b15b0 0x7fee841e8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee8409b8b0 con 0x7fee840b2a20 2024-04-25T14:03:03.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee8406d660 con 0x7fee840b2610 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee8409ba30 con 0x7fee840b15b0 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40640/0 (socket says 172.21.15.150:40640) 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 -- 172.21.15.150:0/218838411 learned_addr learned my addr 172.21.15.150:0/218838411 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee83fff640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee840b15b0 0x7fee841e8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 -- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee840b15b0 msgr2=0x7fee841e8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee88a7e640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee840b2610 0x7fee840b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee840b15b0 0x7fee841e8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 -- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee840b2610 msgr2=0x7fee840b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee840b2610 0x7fee840b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 -- 172.21.15.150:0/218838411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee840b09e0 con 0x7fee840b2a20 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8927f640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fee7c02b070 tx=0x7fee7c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c045070 con 0x7fee840b2a20 2024-04-25T14:03:03.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7c003990 con 0x7fee840b2a20 2024-04-25T14:03:03.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee841e9620 con 0x7fee840b2a20 2024-04-25T14:03:03.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.411+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7c009be0 con 0x7fee840b2a20 2024-04-25T14:03:03.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.412+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee841e7de0 con 0x7fee840b2a20 2024-04-25T14:03:03.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.412+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee84076a40 con 0x7fee840b2a20 2024-04-25T14:03:03.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.413+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee7c02e830 con 0x7fee840b2a20 2024-04-25T14:03:03.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.413+0000 7fee817fa640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 0x7fee700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.414+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fee7c0b5b00 con 0x7fee840b2a20 2024-04-25T14:03:03.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.414+0000 7fee88a7e640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 0x7fee700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.414+0000 7fee88a7e640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 0x7fee700769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee740012b0 tx=0x7fee74009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.418+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee7c03d070 con 0x7fee840b2a20 2024-04-25T14:03:03.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.593+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fee840b1b80 con 0x7fee840b2a20 2024-04-25T14:03:03.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.593+0000 7fee817fa640 1 -- 172.21.15.150:0/218838411 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fee7c032030 con 0x7fee840b2a20 2024-04-25T14:03:03.594 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 msgr2=0x7fee700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 0x7fee700769c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fee740012b0 tx=0x7fee74009040 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 msgr2=0x7fee841e90e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fee7c02b070 tx=0x7fee7c004ab0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 shutdown_connections 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee700744b0 0x7fee700769c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee840b2a20 0x7fee841e90e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee840b15b0 0x7fee841e8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 --2- 172.21.15.150:0/218838411 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee840b2610 0x7fee840b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 >> 172.21.15.150:0/218838411 conn(0x7fee840b7680 msgr2=0x7fee8415d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 shutdown_connections 2024-04-25T14:03:03.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.595+0000 7fee8ace2640 1 -- 172.21.15.150:0/218838411 wait complete. 2024-04-25T14:03:03.607 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:03:03.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.716+0000 7fc3364f2640 1 Processor -- start 2024-04-25T14:03:03.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.716+0000 7fc3364f2640 1 -- start start 2024-04-25T14:03:03.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc3364f2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc3300b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc3364f2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3300b9860 con 0x7fc330164070 2024-04-25T14:03:03.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc3364f2640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3300b92a0 con 0x7fc3300d2dc0 2024-04-25T14:03:03.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc3364f2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3300b9580 con 0x7fc3300b67b0 2024-04-25T14:03:03.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc32f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc330164070 0x7fc330164480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34998/0 (socket says 172.21.15.150:34998) 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc32f7fe640 1 -- 172.21.15.150:0/2641812360 learned_addr learned my addr 172.21.15.150:0/2641812360 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc3300b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2557194358 0 0) 0x7fc3300b9860 con 0x7fc330164070 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.717+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc318003520 con 0x7fc330164070 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3426005800 0 0) 0x7fc3300b92a0 con 0x7fc3300d2dc0 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3300b9860 con 0x7fc3300d2dc0 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1383948537 0 0) 0x7fc318003520 con 0x7fc330164070 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 msgr2=0x7fc3300b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc3300b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 >> v1:172.21.15.187:6789/0 conn(0x7fc3300d2dc0 legacy=0x7fc3301604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3300d15d0 con 0x7fc330164070 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fc324001670 con 0x7fc330164070 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 >> v1:172.21.15.150:6789/0 conn(0x7fc330164070 legacy=0x7fc330164480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc318003b70 0x7fc318003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc318004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318004d40 0x7fc318005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3300b9580 con 0x7fc318003b70 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc31800bbd0 con 0x7fc318004d40 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc31800bdd0 con 0x7fc3300b67b0 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc318003b70 0x7fc318003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32f7fe640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc318004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc334a8f640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318004d40 0x7fc318005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 -- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 msgr2=0x7fc318004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3300b67b0 0x7fc318004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 -- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318004d40 msgr2=0x7fc318005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318004d40 0x7fc318005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc32ffff640 1 -- 172.21.15.150:0/2641812360 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc318003520 con 0x7fc318003b70 2024-04-25T14:03:03.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.718+0000 7fc334a8f640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318004d40 0x7fc318005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.719+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc318003b70 0x7fc318003fc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc33008ca70 tx=0x7fc318005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a6bd2188faf0a66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.719+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320036070 con 0x7fc318003b70 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.719+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc320031400 con 0x7fc318003b70 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.719+0000 7fc32effd640 1 -- 172.21.15.150:0/2641812360 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320009690 con 0x7fc318003b70 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 -- 172.21.15.150:0/2641812360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc318003b70 msgr2=0x7fc318003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2641812360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc318003b70 0x7fc318003fc0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc33008ca70 tx=0x7fc318005830 comp rx=0 tx=0).stop 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc334a8f640 1 -- 172.21.15.150:0/2641812360 reap_dead start 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 -- 172.21.15.150:0/2641812360 shutdown_connections 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 -- 172.21.15.150:0/2641812360 >> 172.21.15.150:0/2641812360 conn(0x7fc3300d3960 msgr2=0x7fc3300d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 -- 172.21.15.150:0/2641812360 shutdown_connections 2024-04-25T14:03:03.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.720+0000 7fc3364f2640 1 -- 172.21.15.150:0/2641812360 wait complete. 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.722+0000 7fc3364f2640 1 Processor -- start 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.722+0000 7fc3364f2640 1 -- start start 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318003b70 0x7fc33015f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc33014a740 0x7fc33014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3300b8fe0 con 0x7fc3300b34a0 2024-04-25T14:03:03.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3300b8e60 con 0x7fc318003b70 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc3364f2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3300b6bc0 con 0x7fc33014a740 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40648/0 (socket says 172.21.15.150:40648) 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 -- 172.21.15.150:0/2295987309 learned_addr learned my addr 172.21.15.150:0/2295987309 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc334a8f640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc33014a740 0x7fc33014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318003b70 0x7fc33015f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 -- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc33014a740 msgr2=0x7fc33014ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc33014a740 0x7fc33014ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 -- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318003b70 msgr2=0x7fc33015f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318003b70 0x7fc33015f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.723+0000 7fc32f7fe640 1 -- 172.21.15.150:0/2295987309 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc33014b090 con 0x7fc3300b34a0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc32f7fe640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc32400e760 tx=0x7fc32400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32400c6c0 con 0x7fc3300b34a0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc324018070 con 0x7fc3300b34a0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc32400caa0 con 0x7fc3300b34a0 2024-04-25T14:03:03.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc33014c480 con 0x7fc3300b34a0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.724+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc33014ca40 con 0x7fc3300b34a0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.725+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc33014c1f0 con 0x7fc3300b34a0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.726+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc32400cc40 con 0x7fc3300b34a0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.726+0000 7fc32cff9640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 0x7fc31c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.726+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fc3240953b0 con 0x7fc3300b34a0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.726+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 0x7fc31c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:03.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.727+0000 7fc32ffff640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 0x7fc31c0769f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc320033000 tx=0x7fc320008030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:03.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.731+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc33014b090 con 0x7fc3300b34a0 2024-04-25T14:03:03.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.917+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc330076a40 con 0x7fc3300b34a0 2024-04-25T14:03:03.918 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.918+0000 7fc32cff9640 1 -- 172.21.15.150:0/2295987309 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7fc32405f740 con 0x7fc3300b34a0 2024-04-25T14:03:03.918 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:03:03.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 msgr2=0x7fc31c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 0x7fc31c0769f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc320033000 tx=0x7fc320008030 comp rx=0 tx=0).stop 2024-04-25T14:03:03.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 msgr2=0x7fc3300b38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:03.920 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc32400e760 tx=0x7fc32400ec20 comp rx=0 tx=0).stop 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 shutdown_connections 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc31c0744e0 0x7fc31c0769f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc33014a740 0x7fc33014ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3300b34a0 0x7fc3300b38b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 --2- 172.21.15.150:0/2295987309 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc318003b70 0x7fc33015f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 >> 172.21.15.150:0/2295987309 conn(0x7fc3300d3960 msgr2=0x7fc33015be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 shutdown_connections 2024-04-25T14:03:03.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:03.920+0000 7fc3364f2640 1 -- 172.21.15.150:0/2295987309 wait complete. 2024-04-25T14:03:03.931 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-04-25T14:03:03.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:03.932 DEBUG:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-04-25T14:03:03.962 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:03:03.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:03.963 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-25T14:03:04.027 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:04.027 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:04.043 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:04.043 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:04.098 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:04.098 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:04.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:04.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.301+0000 7f7c8ec9f640 1 Processor -- start 2024-04-25T14:03:04.302 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.301+0000 7f7c8ec9f640 1 -- start start 2024-04-25T14:03:04.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ec9f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c88160240 0x7f7c880d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ec9f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88164910 con 0x7f7c880b5ab0 2024-04-25T14:03:04.303 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ec9f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c880cfdb0 con 0x7f7c880b2ca0 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ec9f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7c880b80b0 con 0x7f7c88160240 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ca3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7c88160240 0x7f7c880d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8d23c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7c880b5ab0 0x7f7c880b8540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35012/0 (socket says 172.21.15.150:35012) 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8d23c640 1 -- 172.21.15.150:0/3782561259 learned_addr learned my addr 172.21.15.150:0/3782561259 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ca3b640 1 -- 172.21.15.150:0/3782561259 >> v1:172.21.15.187:6789/0 conn(0x7f7c880b2ca0 legacy=0x7f7c880b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ca3b640 1 -- 172.21.15.150:0/3782561259 >> v1:172.21.15.150:6789/0 conn(0x7f7c880b5ab0 legacy=0x7f7c880b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c877fe640 1 -- 172.21.15.150:0/3782561259 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240399907 0 0) 0x7f7c88164910 con 0x7f7c880b5ab0 2024-04-25T14:03:04.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.303+0000 7f7c8ca3b640 1 -- 172.21.15.150:0/3782561259 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c880d0090 con 0x7f7c88160240 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.304+0000 7f7c8ca3b640 1 --2- 172.21.15.150:0/3782561259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c880d2d60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c780097a0 tx=0x7f7c7802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7eacfc2f24ac60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.304+0000 7f7c877fe640 1 -- 172.21.15.150:0/3782561259 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78037070 con 0x7f7c88160240 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.304+0000 7f7c877fe640 1 -- 172.21.15.150:0/3782561259 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c78032440 con 0x7f7c88160240 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.304+0000 7f7c877fe640 1 -- 172.21.15.150:0/3782561259 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c78032810 con 0x7f7c88160240 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/3782561259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 msgr2=0x7f7c880d2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/3782561259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c880d2d60 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f7c780097a0 tx=0x7f7c7802a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/3782561259 shutdown_connections 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/3782561259 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c880d2d60 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/3782561259 >> 172.21.15.150:0/3782561259 conn(0x7f7c880d03a0 msgr2=0x7f7c880d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/3782561259 shutdown_connections 2024-04-25T14:03:04.306 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.305+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/3782561259 wait complete. 2024-04-25T14:03:04.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.307+0000 7f7c8ec9f640 1 Processor -- start 2024-04-25T14:03:04.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.307+0000 7f7c8ec9f640 1 -- start start 2024-04-25T14:03:04.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c880b2ca0 0x7f7c880cc340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c880b5ab0 0x7f7c88162270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880b8dd0 con 0x7f7c88160240 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7c880b8c50 con 0x7f7c880b2ca0 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ec9f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7c880b8f50 con 0x7f7c880b5ab0 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8d23c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8d23c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40666/0 (socket says 172.21.15.150:40666) 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8d23c640 1 -- 172.21.15.150:0/661893442 learned_addr learned my addr 172.21.15.150:0/661893442 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c8ca3b640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c880b2ca0 0x7f7c880cc340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.308+0000 7f7c87fff640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c880b5ab0 0x7f7c88162270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 -- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c880b5ab0 msgr2=0x7f7c88162270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c880b5ab0 0x7f7c88162270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 -- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c880b2ca0 msgr2=0x7f7c880cc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c880b2ca0 0x7f7c880cc340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 -- 172.21.15.150:0/661893442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c880cc880 con 0x7f7c88160240 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8d23c640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000caf0 tx=0x7f7c80002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c8000ab70 con 0x7f7c88160240 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c8000ad10 con 0x7f7c88160240 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c88158800 con 0x7f7c88160240 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c80012640 con 0x7f7c88160240 2024-04-25T14:03:04.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.309+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c88158dc0 con 0x7f7c88160240 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.310+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c8808ccb0 con 0x7f7c88160240 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.311+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7c800127e0 con 0x7f7c88160240 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.311+0000 7f7c857fa640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 0x7f7c740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.311+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f7c80095aa0 con 0x7f7c88160240 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.311+0000 7f7c8ca3b640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 0x7f7c740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.312+0000 7f7c8ca3b640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 0x7f7c740769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c78034000 tx=0x7f7c7802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.316+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7c8808ccb0 con 0x7f7c88160240 2024-04-25T14:03:04.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.498+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7c88076a40 con 0x7f7c88160240 2024-04-25T14:03:04.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.499+0000 7f7c857fa640 1 -- 172.21.15.150:0/661893442 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7f7c8005f7e0 con 0x7f7c88160240 2024-04-25T14:03:04.499 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:04.499 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:04.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 msgr2=0x7f7c740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 0x7f7c740769c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7c78034000 tx=0x7f7c7802e000 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 msgr2=0x7f7c881627b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7c8000caf0 tx=0x7f7c80002190 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 shutdown_connections 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7c740744b0 0x7f7c740769c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7c88160240 0x7f7c881627b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7c880b5ab0 0x7f7c88162270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 --2- 172.21.15.150:0/661893442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7c880b2ca0 0x7f7c880cc340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 >> 172.21.15.150:0/661893442 conn(0x7f7c880d03a0 msgr2=0x7f7c88163b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 shutdown_connections 2024-04-25T14:03:04.502 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.501+0000 7f7c8ec9f640 1 -- 172.21.15.150:0/661893442 wait complete. 2024-04-25T14:03:04.513 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:03:04.513 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.625+0000 7f6e00849640 1 Processor -- start 2024-04-25T14:03:04.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.625+0000 7f6e00849640 1 -- start start 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6e00849640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dfc160770 0x7f6dfc15a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6e00849640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 0x7f6dfc09b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6e00849640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dfc160ce0 con 0x7f6dfc06f6e0 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6e00849640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6dfc160fc0 con 0x7f6dfc160770 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6e00849640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6dfc1612a0 con 0x7f6dfc06f2d0 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dfc160770 0x7f6dfc15a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6dfc160770 0x7f6dfc15a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40674/0 (socket says 172.21.15.150:40674) 2024-04-25T14:03:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 -- 172.21.15.150:0/3843443509 learned_addr learned my addr 172.21.15.150:0/3843443509 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6df9d9b640 1 --2- 172.21.15.150:0/3843443509 >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 0x7f6dfc09b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 -- 172.21.15.150:0/3843443509 >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 msgr2=0x7f6dfc09b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 --2- 172.21.15.150:0/3843443509 >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 0x7f6dfc09b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 -- 172.21.15.150:0/3843443509 >> v1:172.21.15.187:6789/0 conn(0x7f6dfc06f6e0 legacy=0x7f6dfc163d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6dfa59c640 1 -- 172.21.15.150:0/3843443509 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc161580 con 0x7f6dfc160770 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.627+0000 7f6df9d9b640 1 --2- 172.21.15.150:0/3843443509 >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 0x7f6dfc09b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.628+0000 7f6dfa59c640 1 --2- 172.21.15.150:0/3843443509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc15a850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc06f090 tx=0x7f6dec02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e44eb3cfc4c6e4cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.628+0000 7f6df959a640 1 -- 172.21.15.150:0/3843443509 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec037070 con 0x7f6dfc160770 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.628+0000 7f6df959a640 1 -- 172.21.15.150:0/3843443509 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dec032440 con 0x7f6dfc160770 2024-04-25T14:03:04.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.628+0000 7f6df959a640 1 -- 172.21.15.150:0/3843443509 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dec032810 con 0x7f6dfc160770 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 -- 172.21.15.150:0/3843443509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 msgr2=0x7f6dfc15a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 --2- 172.21.15.150:0/3843443509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc15a850 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc06f090 tx=0x7f6dec02a230 comp rx=0 tx=0).stop 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 -- 172.21.15.150:0/3843443509 shutdown_connections 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 --2- 172.21.15.150:0/3843443509 >> v2:172.21.15.187:3300/0 conn(0x7f6dfc06f2d0 0x7f6dfc09b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 --2- 172.21.15.150:0/3843443509 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc15a850 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 -- 172.21.15.150:0/3843443509 >> 172.21.15.150:0/3843443509 conn(0x7f6dfc0b7680 msgr2=0x7f6dfc0b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 -- 172.21.15.150:0/3843443509 shutdown_connections 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.629+0000 7f6e00849640 1 -- 172.21.15.150:0/3843443509 wait complete. 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.631+0000 7f6e00849640 1 Processor -- start 2024-04-25T14:03:04.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.631+0000 7f6e00849640 1 -- start start 2024-04-25T14:03:04.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dfc06f2d0 0x7f6dfc1462c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dfc09f9b0 0x7f6dfc09fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc09bb60 con 0x7f6dfc160770 2024-04-25T14:03:04.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6dfc09b9e0 con 0x7f6dfc09f9b0 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6e00849640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6dfc09bce0 con 0x7f6dfc06f2d0 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40680/0 (socket says 172.21.15.150:40680) 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 -- 172.21.15.150:0/1336323791 learned_addr learned my addr 172.21.15.150:0/1336323791 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6dfa59c640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dfc06f2d0 0x7f6dfc1462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6dfad9d640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dfc09f9b0 0x7f6dfc09fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 -- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dfc06f2d0 msgr2=0x7f6dfc1462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dfc06f2d0 0x7f6dfc1462c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 -- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dfc09f9b0 msgr2=0x7f6dfc09fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dfc09f9b0 0x7f6dfc09fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.632+0000 7f6df9d9b640 1 -- 172.21.15.150:0/1336323791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dfc146800 con 0x7f6dfc160770 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6df9d9b640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6df000c7f0 tx=0x7f6df000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df0015070 con 0x7f6dfc160770 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6df0007820 con 0x7f6dfc160770 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dfc0a0300 con 0x7f6dfc160770 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6df0007bf0 con 0x7f6dfc160770 2024-04-25T14:03:04.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.633+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dfc09d550 con 0x7f6dfc160770 2024-04-25T14:03:04.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.634+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dfc076a40 con 0x7f6dfc160770 2024-04-25T14:03:04.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.634+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6df0018070 con 0x7f6dfc160770 2024-04-25T14:03:04.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.635+0000 7f6deaffd640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 0x7f6ddc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.635+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6df00955b0 con 0x7f6dfc160770 2024-04-25T14:03:04.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.635+0000 7f6dfa59c640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 0x7f6ddc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.635+0000 7f6dfa59c640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 0x7f6ddc0769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6dec034000 tx=0x7f6dec02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.640+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6dfc0a0300 con 0x7f6dfc160770 2024-04-25T14:03:04.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.812+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6dfc0b2600 con 0x7f6dfc160770 2024-04-25T14:03:04.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.813+0000 7f6deaffd640 1 -- 172.21.15.150:0/1336323791 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f6dfc146800 con 0x7f6dfc160770 2024-04-25T14:03:04.813 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 msgr2=0x7f6ddc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 0x7f6ddc0769c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6dec034000 tx=0x7f6dec02e000 comp rx=0 tx=0).stop 2024-04-25T14:03:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 msgr2=0x7f6dfc09c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f6df000c7f0 tx=0x7f6df000ccb0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.815 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 shutdown_connections 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6ddc0744b0 0x7f6ddc0769c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6dfc09f9b0 0x7f6dfc09fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6dfc160770 0x7f6dfc09c7d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 --2- 172.21.15.150:0/1336323791 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6dfc06f2d0 0x7f6dfc1462c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 >> 172.21.15.150:0/1336323791 conn(0x7f6dfc0b7680 msgr2=0x7f6dfc0aff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 shutdown_connections 2024-04-25T14:03:04.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.815+0000 7f6e00849640 1 -- 172.21.15.150:0/1336323791 wait complete. 2024-04-25T14:03:04.827 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:03:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.936+0000 7f13b6277640 1 Processor -- start 2024-04-25T14:03:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.936+0000 7f13b6277640 1 -- start start 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13b006c570 0x7f13b006daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b00d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 0x7f13b00d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f13b0160740 con 0x7f13b006c570 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f13b00d5200 con 0x7f13b00d48b0 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b6277640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13b006ca50 con 0x7f13b0160330 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13b006c570 0x7f13b006daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f13b006c570 0x7f13b006daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40690/0 (socket says 172.21.15.150:40690) 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 -- 172.21.15.150:0/34417624 learned_addr learned my addr 172.21.15.150:0/34417624 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b4814640 1 --2- 172.21.15.150:0/34417624 >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 0x7f13b00d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13af7fe640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b00d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 -- 172.21.15.150:0/34417624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 msgr2=0x7f13b00d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:04.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b00d3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 -- 172.21.15.150:0/34417624 >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 msgr2=0x7f13b00d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:04.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 --2- 172.21.15.150:0/34417624 >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 0x7f13b00d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13affff640 1 -- 172.21.15.150:0/34417624 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b006cd30 con 0x7f13b006c570 2024-04-25T14:03:04.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.938+0000 7f13b4814640 1 --2- 172.21.15.150:0/34417624 >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 0x7f13b00d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:04.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.939+0000 7f13affff640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b006daf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f13b00d3750 tx=0x7f13a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aa6d39f528590a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.939+0000 7f13aeffd640 1 -- 172.21.15.150:0/34417624 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0037070 con 0x7f13b006c570 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.939+0000 7f13aeffd640 1 -- 172.21.15.150:0/34417624 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a0032440 con 0x7f13b006c570 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.939+0000 7f13aeffd640 1 -- 172.21.15.150:0/34417624 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0032810 con 0x7f13b006c570 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 -- 172.21.15.150:0/34417624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 msgr2=0x7f13b006daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b006daf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f13b00d3750 tx=0x7f13a002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 -- 172.21.15.150:0/34417624 shutdown_connections 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 --2- 172.21.15.150:0/34417624 >> v2:172.21.15.187:3300/0 conn(0x7f13b00d48b0 0x7f13b00d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b00d3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 --2- 172.21.15.150:0/34417624 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b006daf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 -- 172.21.15.150:0/34417624 >> 172.21.15.150:0/34417624 conn(0x7f13b00d63f0 msgr2=0x7f13b00d2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 -- 172.21.15.150:0/34417624 shutdown_connections 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.940+0000 7f13b6277640 1 -- 172.21.15.150:0/34417624 wait complete. 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.942+0000 7f13b6277640 1 Processor -- start 2024-04-25T14:03:04.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.942+0000 7f13b6277640 1 -- start start 2024-04-25T14:03:04.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 0x7f13b00ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.944 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b0159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f13b00d44d0 con 0x7f13b006c570 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f13b006d2f0 con 0x7f13b00d48b0 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b6277640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f13b00d4650 con 0x7f13b0160330 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40694/0 (socket says 172.21.15.150:40694) 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 -- 172.21.15.150:0/840173564 learned_addr learned my addr 172.21.15.150:0/840173564 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13af7fe640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 0x7f13b00ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13b4814640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b0159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 -- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 msgr2=0x7f13b0159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b0159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 -- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 msgr2=0x7f13b00ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 0x7f13b00ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13affff640 1 -- 172.21.15.150:0/840173564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13b00cd830 con 0x7f13b006c570 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.943+0000 7f13af7fe640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 0x7f13b00ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:04.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13affff640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000c00 tx=0x7f13a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0041020 con 0x7f13b006c570 2024-04-25T14:03:04.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13a00306c0 con 0x7f13b006c570 2024-04-25T14:03:04.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13b0160a90 con 0x7f13b006c570 2024-04-25T14:03:04.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0037040 con 0x7f13b006c570 2024-04-25T14:03:04.946 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.944+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13b0161050 con 0x7f13b006c570 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.945+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13b0076a40 con 0x7f13b006c570 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.945+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f13a0044020 con 0x7f13b006c570 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.946+0000 7f13acff9640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 0x7f139c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.946+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f13a00b5d30 con 0x7f13b006c570 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.946+0000 7f13af7fe640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 0x7f139c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:04.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.946+0000 7f13af7fe640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 0x7f139c0769f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:04.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:04.950+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f13a004b030 con 0x7f13b006c570 2024-04-25T14:03:05.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.125+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f13b015b7c0 con 0x7f13b006c570 2024-04-25T14:03:05.125 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.125+0000 7f13acff9640 1 -- 172.21.15.150:0/840173564 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f13a00801e0 con 0x7f13b006c570 2024-04-25T14:03:05.126 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 msgr2=0x7f139c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 0x7f139c0769f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f13a4006fd0 tx=0x7f13a4008040 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 msgr2=0x7f13b00cf440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f13a0000c00 tx=0x7f13a00046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 shutdown_connections 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f139c0744e0 0x7f139c0769f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f13b0160330 0x7f13b0159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f13b00d48b0 0x7f13b00ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 --2- 172.21.15.150:0/840173564 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f13b006c570 0x7f13b00cf440 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 >> 172.21.15.150:0/840173564 conn(0x7f13b00d63f0 msgr2=0x7f13b015f4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 shutdown_connections 2024-04-25T14:03:05.128 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.127+0000 7f13b6277640 1 -- 172.21.15.150:0/840173564 wait complete. 2024-04-25T14:03:05.139 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:03:05.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.251+0000 7fa88a267640 1 Processor -- start 2024-04-25T14:03:05.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.251+0000 7fa88a267640 1 -- start start 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.252+0000 7fa88a267640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 0x7fa88406f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.252+0000 7fa88a267640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa88406d510 0x7fa8840cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.252+0000 7fa88a267640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88406fef0 con 0x7fa8840cd030 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.252+0000 7fa88a267640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa88406f520 con 0x7fa88406d510 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.252+0000 7fa88a267640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa88415c920 con 0x7fa88415c510 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa8837fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa8840cd030 0x7fa88406ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35020/0 (socket says 172.21.15.150:35020) 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa8837fe640 1 -- 172.21.15.150:0/2356921368 learned_addr learned my addr 172.21.15.150:0/2356921368 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa883fff640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 0x7fa88406f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa888804640 1 --2- 172.21.15.150:0/2356921368 >> v2:172.21.15.150:3300/0 conn(0x7fa88406d510 0x7fa8840cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa882ffd640 1 -- 172.21.15.150:0/2356921368 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3867288178 0 0) 0x7fa878001110 con 0x7fa8840cd030 2024-04-25T14:03:05.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa882ffd640 1 -- 172.21.15.150:0/2356921368 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa86c003520 con 0x7fa8840cd030 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa888804640 1 -- 172.21.15.150:0/2356921368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 msgr2=0x7fa88406f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa888804640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 0x7fa88406f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa888804640 1 -- 172.21.15.150:0/2356921368 >> v1:172.21.15.150:6789/0 conn(0x7fa8840cd030 legacy=0x7fa88406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa888804640 1 -- 172.21.15.150:0/2356921368 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8840cdaa0 con 0x7fa88406d510 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.253+0000 7fa883fff640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 0x7fa88406f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.254+0000 7fa888804640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88406d510 0x7fa8840cd440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009d90 tx=0x7fa87c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79447e3d1a9f1f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.254+0000 7fa882ffd640 1 -- 172.21.15.150:0/2356921368 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c007740 con 0x7fa88406d510 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.254+0000 7fa882ffd640 1 -- 172.21.15.150:0/2356921368 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c0078e0 con 0x7fa88406d510 2024-04-25T14:03:05.255 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.254+0000 7fa882ffd640 1 -- 172.21.15.150:0/2356921368 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c007cb0 con 0x7fa88406d510 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 -- 172.21.15.150:0/2356921368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88406d510 msgr2=0x7fa8840cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88406d510 0x7fa8840cd440 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fa87c009d90 tx=0x7fa87c02a680 comp rx=0 tx=0).stop 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 -- 172.21.15.150:0/2356921368 shutdown_connections 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88406d510 0x7fa8840cd440 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 --2- 172.21.15.150:0/2356921368 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88415c510 0x7fa88406f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 -- 172.21.15.150:0/2356921368 >> 172.21.15.150:0/2356921368 conn(0x7fa8840ce390 msgr2=0x7fa8840ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 -- 172.21.15.150:0/2356921368 shutdown_connections 2024-04-25T14:03:05.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.255+0000 7fa88a267640 1 -- 172.21.15.150:0/2356921368 wait complete. 2024-04-25T14:03:05.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.257+0000 7fa88a267640 1 Processor -- start 2024-04-25T14:03:05.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.257+0000 7fa88a267640 1 -- start start 2024-04-25T14:03:05.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8840cd030 0x7fa88409b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88414f2e0 0x7fa88415eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa88409f720 con 0x7fa88415c510 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa88406f1b0 con 0x7fa8840cd030 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa88a267640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa88409f8a0 con 0x7fa88414f2e0 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40712/0 (socket says 172.21.15.150:40712) 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 -- 172.21.15.150:0/310929516 learned_addr learned my addr 172.21.15.150:0/310929516 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa888804640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88414f2e0 0x7fa88415eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 -- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88414f2e0 msgr2=0x7fa88415eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88414f2e0 0x7fa88415eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 -- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8840cd030 msgr2=0x7fa88409b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8840cd030 0x7fa88409b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.258+0000 7fa8837fe640 1 -- 172.21.15.150:0/310929516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88409bc90 con 0x7fa88415c510 2024-04-25T14:03:05.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa8837fe640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa884070d20 tx=0x7fa87800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878017070 con 0x7fa88415c510 2024-04-25T14:03:05.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8780058b0 con 0x7fa88415c510 2024-04-25T14:03:05.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa878005c80 con 0x7fa88415c510 2024-04-25T14:03:05.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88409bf70 con 0x7fa88415c510 2024-04-25T14:03:05.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.259+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa88414dcf0 con 0x7fa88415c510 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.260+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884076a40 con 0x7fa88415c510 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.261+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa87801a070 con 0x7fa88415c510 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.261+0000 7fa880ff9640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 0x7fa8700769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.261+0000 7fa883fff640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 0x7fa8700769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.261+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa8780946a0 con 0x7fa88415c510 2024-04-25T14:03:05.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.262+0000 7fa883fff640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 0x7fa8700769f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa874006e80 tx=0x7fa874006dd0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.266+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa88406fef0 con 0x7fa88415c510 2024-04-25T14:03:05.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.445+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa88409ba20 con 0x7fa88415c510 2024-04-25T14:03:05.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.445+0000 7fa880ff9640 1 -- 172.21.15.150:0/310929516 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7fa878005e20 con 0x7fa88415c510 2024-04-25T14:03:05.445 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:03:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 msgr2=0x7fa8700769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 0x7fa8700769f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa874006e80 tx=0x7fa874006dd0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 msgr2=0x7fa88414c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa884070d20 tx=0x7fa87800cb90 comp rx=0 tx=0).stop 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 shutdown_connections 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa8700744e0 0x7fa8700769f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa88414f2e0 0x7fa88415eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa88415c510 0x7fa88414c030 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 --2- 172.21.15.150:0/310929516 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8840cd030 0x7fa88409b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 >> 172.21.15.150:0/310929516 conn(0x7fa8840ce390 msgr2=0x7fa88406d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 shutdown_connections 2024-04-25T14:03:05.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.447+0000 7fa88a267640 1 -- 172.21.15.150:0/310929516 wait complete. 2024-04-25T14:03:05.459 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:03:05.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.569+0000 7fa42051f640 1 Processor -- start 2024-04-25T14:03:05.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.569+0000 7fa42051f640 1 -- start start 2024-04-25T14:03:05.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa42051f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 0x7fa4180d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa42051f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa42051f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4180d15d0 con 0x7fa4180cf3d0 2024-04-25T14:03:05.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa42051f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa4180ce440 con 0x7fa418070830 2024-04-25T14:03:05.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa42051f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4180d1870 con 0x7fa41815e360 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa41eabc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.570+0000 7fa41daba640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 0x7fa4180d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:45652/0 (socket says 172.21.15.150:45652) 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 -- 172.21.15.150:0/4245415675 learned_addr learned my addr 172.21.15.150:0/4245415675 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41e2bb640 1 --1- 172.21.15.150:0/4245415675 >> v1:172.21.15.187:6789/0 conn(0x7fa4180cf3d0 0x7fa418070120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:34276/0 (socket says 172.21.15.150:34276) 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 -- 172.21.15.150:0/4245415675 >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 msgr2=0x7fa4180d2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 --2- 172.21.15.150:0/4245415675 >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 0x7fa4180d2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 -- 172.21.15.150:0/4245415675 >> v1:172.21.15.187:6789/0 conn(0x7fa4180cf3d0 legacy=0x7fa418070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41eabc640 1 -- 172.21.15.150:0/4245415675 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4180d1b50 con 0x7fa41815e360 2024-04-25T14:03:05.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.571+0000 7fa41daba640 1 --2- 172.21.15.150:0/4245415675 >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 0x7fa4180d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.572+0000 7fa41eabc640 1 --2- 172.21.15.150:0/4245415675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa410009870 tx=0x7fa41002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4da7c3454ba2f80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.572+0000 7fa41d2b9640 1 -- 172.21.15.150:0/4245415675 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410037070 con 0x7fa41815e360 2024-04-25T14:03:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.572+0000 7fa41d2b9640 1 -- 172.21.15.150:0/4245415675 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa410032440 con 0x7fa41815e360 2024-04-25T14:03:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.572+0000 7fa41d2b9640 1 -- 172.21.15.150:0/4245415675 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410032810 con 0x7fa41815e360 2024-04-25T14:03:05.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 -- 172.21.15.150:0/4245415675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 msgr2=0x7fa4180d25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 --2- 172.21.15.150:0/4245415675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa410009870 tx=0x7fa41002a270 comp rx=0 tx=0).stop 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 -- 172.21.15.150:0/4245415675 shutdown_connections 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 --2- 172.21.15.150:0/4245415675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa41815e360 0x7fa4180d25b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 --2- 172.21.15.150:0/4245415675 >> v2:172.21.15.187:3300/0 conn(0x7fa418070830 0x7fa4180d2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 -- 172.21.15.150:0/4245415675 >> 172.21.15.150:0/4245415675 conn(0x7fa4180d3960 msgr2=0x7fa4180d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 -- 172.21.15.150:0/4245415675 shutdown_connections 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.573+0000 7fa42051f640 1 -- 172.21.15.150:0/4245415675 wait complete. 2024-04-25T14:03:05.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.574+0000 7fa42051f640 1 Processor -- start 2024-04-25T14:03:05.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.574+0000 7fa42051f640 1 -- start start 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4180cf3d0 0x7fa418154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 0x7fa41815e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa418070eb0 con 0x7fa4181f4490 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa418070d30 con 0x7fa41815e360 2024-04-25T14:03:05.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa42051f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa4180cdfb0 con 0x7fa4180cf3d0 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40726/0 (socket says 172.21.15.150:40726) 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 -- 172.21.15.150:0/1344624634 learned_addr learned my addr 172.21.15.150:0/1344624634 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41e2bb640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4180cf3d0 0x7fa418154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41daba640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 0x7fa41815e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 -- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4180cf3d0 msgr2=0x7fa418154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4180cf3d0 0x7fa418154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 -- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 msgr2=0x7fa41815e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 0x7fa41815e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 -- 172.21.15.150:0/1344624634 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4181f51f0 con 0x7fa4181f4490 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41daba640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 0x7fa41815e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.576+0000 7fa41eabc640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa41002a2b0 tx=0x7fa4100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.577+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410041020 con 0x7fa4181f4490 2024-04-25T14:03:05.578 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.577+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa41003d640 con 0x7fa4181f4490 2024-04-25T14:03:05.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.577+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4180cab90 con 0x7fa4181f4490 2024-04-25T14:03:05.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.577+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410037040 con 0x7fa4181f4490 2024-04-25T14:03:05.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.577+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4180cb150 con 0x7fa4181f4490 2024-04-25T14:03:05.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.578+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa418076a40 con 0x7fa4181f4490 2024-04-25T14:03:05.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.578+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa410044020 con 0x7fa4181f4490 2024-04-25T14:03:05.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.579+0000 7fa40effd640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 0x7fa3fc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.579+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa4100b5d40 con 0x7fa4181f4490 2024-04-25T14:03:05.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.579+0000 7fa41e2bb640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 0x7fa3fc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.579+0000 7fa41e2bb640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 0x7fa3fc0769f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa4140012e0 tx=0x7fa414005b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.583+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa41004b030 con 0x7fa4181f4490 2024-04-25T14:03:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.758+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa41815a600 con 0x7fa4181f4490 2024-04-25T14:03:05.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.758+0000 7fa40effd640 1 -- 172.21.15.150:0/1344624634 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fa410080170 con 0x7fa4181f4490 2024-04-25T14:03:05.758 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:05.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 msgr2=0x7fa3fc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 0x7fa3fc0769f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa4140012e0 tx=0x7fa414005b70 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 msgr2=0x7fa4181f48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa41002a2b0 tx=0x7fa4100046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 shutdown_connections 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3fc0744e0 0x7fa3fc0769f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4181f4490 0x7fa4181f48a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa41815e360 0x7fa41815e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 --2- 172.21.15.150:0/1344624634 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa4180cf3d0 0x7fa418154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 >> 172.21.15.150:0/1344624634 conn(0x7fa4180d3960 msgr2=0x7fa4180cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 shutdown_connections 2024-04-25T14:03:05.761 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.760+0000 7fa42051f640 1 -- 172.21.15.150:0/1344624634 wait complete. 2024-04-25T14:03:05.772 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:03:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.880+0000 7f5c40467640 1 Processor -- start 2024-04-25T14:03:05.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.880+0000 7f5c40467640 1 -- start start 2024-04-25T14:03:05.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c40467640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 0x7f5c38160330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c40467640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c380d48b0 0x7f5c380d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c40467640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c380d5200 con 0x7f5c3806c570 2024-04-25T14:03:05.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c40467640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5c3806df00 con 0x7f5c380d48b0 2024-04-25T14:03:05.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c40467640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3806ca50 con 0x7f5c3806daf0 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c380d48b0 0x7f5c380d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3da02640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 0x7f5c38160330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3e203640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f5c3806c570 0x7f5c380d3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:34286/0 (socket says 172.21.15.150:34286) 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5c380d48b0 0x7f5c380d4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33020/0 (socket says 172.21.15.150:33020) 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3e203640 1 -- 172.21.15.150:0/3659761460 learned_addr learned my addr 172.21.15.150:0/3659761460 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 -- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 msgr2=0x7f5c38160330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 0x7f5c38160330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 -- 172.21.15.150:0/3659761460 >> v1:172.21.15.187:6789/0 conn(0x7f5c3806c570 legacy=0x7f5c380d3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3ea04640 1 -- 172.21.15.150:0/3659761460 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c3806cd30 con 0x7f5c380d48b0 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3da02640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 0x7f5c38160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.882+0000 7f5c3d201640 1 -- 172.21.15.150:0/3659761460 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 724131051 0 0) 0x7f5c34001110 con 0x7f5c3806c570 2024-04-25T14:03:05.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.883+0000 7f5c3ea04640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c380d48b0 0x7f5c380d4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009d90 tx=0x7f5c3002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c27e2f10cab5862 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.884+0000 7f5c3d201640 1 -- 172.21.15.150:0/3659761460 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30007740 con 0x7f5c380d48b0 2024-04-25T14:03:05.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.884+0000 7f5c3d201640 1 -- 172.21.15.150:0/3659761460 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c300078e0 con 0x7f5c380d48b0 2024-04-25T14:03:05.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.884+0000 7f5c3d201640 1 -- 172.21.15.150:0/3659761460 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30007cb0 con 0x7f5c380d48b0 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.884+0000 7f5c40467640 1 -- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c380d48b0 msgr2=0x7f5c380d4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.884+0000 7f5c40467640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c380d48b0 0x7f5c380d4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009d90 tx=0x7f5c3002a660 comp rx=0 tx=0).stop 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 -- 172.21.15.150:0/3659761460 shutdown_connections 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c380d48b0 0x7f5c380d4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 --2- 172.21.15.150:0/3659761460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c3806daf0 0x7f5c38160330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 -- 172.21.15.150:0/3659761460 >> 172.21.15.150:0/3659761460 conn(0x7f5c380d63f0 msgr2=0x7f5c380d2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 -- 172.21.15.150:0/3659761460 shutdown_connections 2024-04-25T14:03:05.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.885+0000 7f5c40467640 1 -- 172.21.15.150:0/3659761460 wait complete. 2024-04-25T14:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.886+0000 7f5c40467640 1 Processor -- start 2024-04-25T14:03:05.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.886+0000 7f5c40467640 1 -- start start 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3806daf0 0x7f5c380ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 0x7f5c381f9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5c380b9310 con 0x7f5c3806c570 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3806d2f0 con 0x7f5c3806daf0 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c40467640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5c380b9490 con 0x7f5c380d48b0 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40736/0 (socket says 172.21.15.150:40736) 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 -- 172.21.15.150:0/4023201621 learned_addr learned my addr 172.21.15.150:0/4023201621 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3ea04640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 0x7f5c381f9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3da02640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3806daf0 0x7f5c380ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 -- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 msgr2=0x7f5c381f9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 0x7f5c381f9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 -- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3806daf0 msgr2=0x7f5c380ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3806daf0 0x7f5c380ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 -- 172.21.15.150:0/4023201621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c380cf500 con 0x7f5c3806c570 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3ea04640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 0x7f5c381f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c3e203640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c430 tx=0x7f5c3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.888+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c340074e0 con 0x7f5c3806c570 2024-04-25T14:03:05.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.889+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c380cf7e0 con 0x7f5c3806c570 2024-04-25T14:03:05.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.889+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c340047e0 con 0x7f5c3806c570 2024-04-25T14:03:05.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.889+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c34004bd0 con 0x7f5c3806c570 2024-04-25T14:03:05.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.889+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c38158b60 con 0x7f5c3806c570 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.890+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c38076a40 con 0x7f5c3806c570 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.890+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5c34011070 con 0x7f5c3806c570 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.891+0000 7f5c2effd640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 0x7f5c1c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.891+0000 7f5c3da02640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 0x7f5c1c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.891+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5c340951e0 con 0x7f5c3806c570 2024-04-25T14:03:05.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.891+0000 7f5c3da02640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 0x7f5c1c0769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5c380d4170 tx=0x7f5c28008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:05.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:05.896+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5c380d5200 con 0x7f5c3806c570 2024-04-25T14:03:06.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.082+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5c3806ef90 con 0x7f5c3806c570 2024-04-25T14:03:06.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.083+0000 7f5c2effd640 1 -- 172.21.15.150:0/4023201621 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f5c3405f610 con 0x7f5c3806c570 2024-04-25T14:03:06.083 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:03:06.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 msgr2=0x7f5c1c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 0x7f5c1c0769f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f5c380d4170 tx=0x7f5c28008040 comp rx=0 tx=0).stop 2024-04-25T14:03:06.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 msgr2=0x7f5c380cb460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5c3400c430 tx=0x7f5c3400c8f0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 shutdown_connections 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5c1c0744e0 0x7f5c1c0769f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5c380d48b0 0x7f5c381f9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5c3806daf0 0x7f5c380ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 --2- 172.21.15.150:0/4023201621 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5c3806c570 0x7f5c380cb460 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 >> 172.21.15.150:0/4023201621 conn(0x7f5c380d63f0 msgr2=0x7f5c380b6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 shutdown_connections 2024-04-25T14:03:06.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.085+0000 7f5c40467640 1 -- 172.21.15.150:0/4023201621 wait complete. 2024-04-25T14:03:06.096 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-04-25T14:03:06.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:06.096 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-04-25T14:03:06.120 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:03:06.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:06.121 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-25T14:03:06.176 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:06.176 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:06.194 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:06.194 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:06.252 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:03:06.252 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:06.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:06.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.419+0000 7fa08b092640 1 Processor -- start 2024-04-25T14:03:06.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.419+0000 7fa08b092640 1 -- start start 2024-04-25T14:03:06.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08b092640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0840d2e90 0x7fa084160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08b092640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0840b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08b092640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0840b5cf0 con 0x7fa0840b2b10 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08b092640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa0840b7db0 con 0x7fa0840d2e90 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08b092640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa084164690 con 0x7fa0840b53a0 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0840d2e90 0x7fa084160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa0840d2e90 0x7fa084160220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40740/0 (socket says 172.21.15.150:40740) 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 -- 172.21.15.150:0/2556757316 learned_addr learned my addr 172.21.15.150:0/2556757316 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa08962f640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0840b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 -- 172.21.15.150:0/2556757316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 msgr2=0x7fa0840b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0840b57b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 -- 172.21.15.150:0/2556757316 >> v1:172.21.15.187:6789/0 conn(0x7fa0840b2b10 legacy=0x7fa0840cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.421+0000 7fa083fff640 1 -- 172.21.15.150:0/2556757316 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa084164bc0 con 0x7fa0840d2e90 2024-04-25T14:03:06.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.422+0000 7fa083fff640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d2e90 0x7fa084160220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa070009870 tx=0x7fa07002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992ff168d066ec07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.422+0000 7fa0837fe640 1 -- 172.21.15.150:0/2556757316 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070037070 con 0x7fa0840d2e90 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.422+0000 7fa0837fe640 1 -- 172.21.15.150:0/2556757316 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07002ae10 con 0x7fa0840d2e90 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.422+0000 7fa0837fe640 1 -- 172.21.15.150:0/2556757316 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070032660 con 0x7fa0840d2e90 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 -- 172.21.15.150:0/2556757316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d2e90 msgr2=0x7fa084160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d2e90 0x7fa084160220 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa070009870 tx=0x7fa07002a100 comp rx=0 tx=0).stop 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 -- 172.21.15.150:0/2556757316 shutdown_connections 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0840b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 --2- 172.21.15.150:0/2556757316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840d2e90 0x7fa084160220 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 -- 172.21.15.150:0/2556757316 >> 172.21.15.150:0/2556757316 conn(0x7fa0840d03a0 msgr2=0x7fa0840d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 -- 172.21.15.150:0/2556757316 shutdown_connections 2024-04-25T14:03:06.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.423+0000 7fa08b092640 1 -- 172.21.15.150:0/2556757316 wait complete. 2024-04-25T14:03:06.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.424+0000 7fa08b092640 1 Processor -- start 2024-04-25T14:03:06.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.424+0000 7fa08b092640 1 -- start start 2024-04-25T14:03:06.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0841579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d2e90 0x7fa084154400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa0841648e0 con 0x7fa0840b3ce0 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa0840b8350 con 0x7fa0840d2e90 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08b092640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa0840b7fe0 con 0x7fa0840b53a0 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40744/0 (socket says 172.21.15.150:40744) 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 -- 172.21.15.150:0/1398012268 learned_addr learned my addr 172.21.15.150:0/1398012268 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa088e2e640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0841579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa083fff640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d2e90 0x7fa084154400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 -- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 msgr2=0x7fa0841579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0841579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 -- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d2e90 msgr2=0x7fa084154400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d2e90 0x7fa084154400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.426+0000 7fa08962f640 1 -- 172.21.15.150:0/1398012268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa084156010 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa08962f640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00cc80 tx=0x7fa07c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c003990 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07c003b50 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa07c004020 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0841562f0 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.427+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa08406f450 con 0x7fa0840b3ce0 2024-04-25T14:03:06.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.428+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa08406ef60 con 0x7fa0840b3ce0 2024-04-25T14:03:06.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.428+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa07c0041c0 con 0x7fa0840b3ce0 2024-04-25T14:03:06.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.429+0000 7fa0817fa640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 0x7fa0680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.429+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa07c095990 con 0x7fa0840b3ce0 2024-04-25T14:03:06.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.429+0000 7fa088e2e640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 0x7fa0680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.429+0000 7fa088e2e640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 0x7fa0680769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078009060 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.433 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.433+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa07c014030 con 0x7fa0840b3ce0 2024-04-25T14:03:06.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.610+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa084076a40 con 0x7fa0840b3ce0 2024-04-25T14:03:06.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.611+0000 7fa0817fa640 1 -- 172.21.15.150:0/1398012268 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fa07c05f760 con 0x7fa0840b3ce0 2024-04-25T14:03:06.611 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:06.612 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 msgr2=0x7fa0680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 0x7fa0680769f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa078006fd0 tx=0x7fa078009060 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 msgr2=0x7fa084162010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa07c00cc80 tx=0x7fa07c00a5c0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 shutdown_connections 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa0680744e0 0x7fa0680769f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa0840b3ce0 0x7fa084162010 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa0840d2e90 0x7fa084154400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 --2- 172.21.15.150:0/1398012268 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa0840b53a0 0x7fa0841579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 >> 172.21.15.150:0/1398012268 conn(0x7fa0840d03a0 msgr2=0x7fa0840d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:06.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 shutdown_connections 2024-04-25T14:03:06.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.613+0000 7fa08b092640 1 -- 172.21.15.150:0/1398012268 wait complete. 2024-04-25T14:03:06.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:03:06.626 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.736+0000 7fe8e5042640 1 Processor -- start 2024-04-25T14:03:06.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.736+0000 7fe8e5042640 1 -- start start 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 0x7fe8e00b2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe8e006f640 0x7fe8e006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 0x7fe8e009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e015e9c0 con 0x7fe8e006f640 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe8e015eca0 con 0x7fe8e006d490 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.737+0000 7fe8e5042640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e015ef80 con 0x7fe8e015e4e0 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe8e006f640 0x7fe8e006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe8e006f640 0x7fe8e006cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40748/0 (socket says 172.21.15.150:40748) 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 -- 172.21.15.150:0/364775302 learned_addr learned my addr 172.21.15.150:0/364775302 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8ded9d640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 0x7fe8e00b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8df59e640 1 --2- 172.21.15.150:0/364775302 >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 0x7fe8e009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 -- 172.21.15.150:0/364775302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 msgr2=0x7fe8e00b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 0x7fe8e00b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 -- 172.21.15.150:0/364775302 >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 msgr2=0x7fe8e009b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 --2- 172.21.15.150:0/364775302 >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 0x7fe8e009b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8de59c640 1 -- 172.21.15.150:0/364775302 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e015f260 con 0x7fe8e006f640 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8ded9d640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 0x7fe8e00b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.738+0000 7fe8df59e640 1 --2- 172.21.15.150:0/364775302 >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 0x7fe8e009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8de59c640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e006cf50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00b5580 tx=0x7fe8d402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f05a1f98183bb29d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8ddd9b640 1 -- 172.21.15.150:0/364775302 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4037070 con 0x7fe8e006f640 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8ddd9b640 1 -- 172.21.15.150:0/364775302 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d4032440 con 0x7fe8e006f640 2024-04-25T14:03:06.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8ddd9b640 1 -- 172.21.15.150:0/364775302 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4032810 con 0x7fe8e006f640 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 -- 172.21.15.150:0/364775302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 msgr2=0x7fe8e006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e006cf50 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe8e00b5580 tx=0x7fe8d402a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 -- 172.21.15.150:0/364775302 shutdown_connections 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 --2- 172.21.15.150:0/364775302 >> v2:172.21.15.187:3300/0 conn(0x7fe8e006d490 0x7fe8e009b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e006cf50 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 --2- 172.21.15.150:0/364775302 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e015e4e0 0x7fe8e00b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 -- 172.21.15.150:0/364775302 >> 172.21.15.150:0/364775302 conn(0x7fe8e00b7680 msgr2=0x7fe8e00b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 -- 172.21.15.150:0/364775302 shutdown_connections 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.739+0000 7fe8e5042640 1 -- 172.21.15.150:0/364775302 wait complete. 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.741+0000 7fe8e5042640 1 Processor -- start 2024-04-25T14:03:06.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.742+0000 7fe8e5042640 1 -- start start 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e006d490 0x7fe8e0157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e015e4e0 0x7fe8e01e8780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e009bb60 con 0x7fe8e006f640 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe8e009b9e0 con 0x7fe8e015e4e0 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8e5042640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8e009bce0 con 0x7fe8e006d490 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40750/0 (socket says 172.21.15.150:40750) 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 -- 172.21.15.150:0/86859973 learned_addr learned my addr 172.21.15.150:0/86859973 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8df59e640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e015e4e0 0x7fe8e01e8780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8ded9d640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e006d490 0x7fe8e0157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 -- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e006d490 msgr2=0x7fe8e0157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e006d490 0x7fe8e0157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 -- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e015e4e0 msgr2=0x7fe8e01e8780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e015e4e0 0x7fe8e01e8780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 -- 172.21.15.150:0/86859973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8e00b09b0 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.743+0000 7fe8de59c640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4000c00 tx=0x7fe8d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.744+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4041070 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.744+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8d403c870 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.744+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8e00b0c90 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.744+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8d4037040 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.744+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8e00b4d90 con 0x7fe8e006f640 2024-04-25T14:03:06.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.745+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8e00b35e0 con 0x7fe8e006f640 2024-04-25T14:03:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.745+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe8d4044070 con 0x7fe8e006f640 2024-04-25T14:03:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.746+0000 7fe8c77fe640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.746+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fe8d40b6090 con 0x7fe8e006f640 2024-04-25T14:03:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.746+0000 7fe8ded9d640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:06.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.746+0000 7fe8ded9d640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006fd0 tx=0x7fe8d0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:06.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.750+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe8d407fdd0 con 0x7fe8e006f640 2024-04-25T14:03:06.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.924+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe8e00b0740 con 0x7fe8e006f640 2024-04-25T14:03:06.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.924+0000 7fe8c77fe640 1 -- 172.21.15.150:0/86859973 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7fe8d40840d0 con 0x7fe8e006f640 2024-04-25T14:03:06.924 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:06.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 msgr2=0x7fe8cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8d0006fd0 tx=0x7fe8d0008040 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 msgr2=0x7fe8e01e8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4000c00 tx=0x7fe8d40046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 shutdown_connections 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8cc0744b0 0x7fe8cc0769c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8e015e4e0 0x7fe8e01e8780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe8e006f640 0x7fe8e01e8240 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 --2- 172.21.15.150:0/86859973 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8e006d490 0x7fe8e0157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 >> 172.21.15.150:0/86859973 conn(0x7fe8e00b7680 msgr2=0x7fe8e015d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 shutdown_connections 2024-04-25T14:03:06.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:06.926+0000 7fe8e5042640 1 -- 172.21.15.150:0/86859973 wait complete. 2024-04-25T14:03:06.938 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:03:07.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.048+0000 7fdc5ce30640 1 Processor -- start 2024-04-25T14:03:07.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.048+0000 7fdc5ce30640 1 -- start start 2024-04-25T14:03:07.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5ce30640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdc580b5ad0 0x7fdc580d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5ce30640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc580d21c0 con 0x7fdc580b2b10 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5ce30640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdc580d24a0 con 0x7fdc5815e010 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5ce30640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdc580d37a0 con 0x7fdc580b5ad0 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdc580b2b10 0x7fdc580cee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35032/0 (socket says 172.21.15.150:35032) 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5659c640 1 -- 172.21.15.150:0/1388094380 learned_addr learned my addr 172.21.15.150:0/1388094380 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc56d9d640 1 --2- 172.21.15.150:0/1388094380 >> v2:172.21.15.187:3300/0 conn(0x7fdc580b5ad0 0x7fdc580d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 249850217 0 0) 0x7fdc580d21c0 con 0x7fdc580b2b10 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc40003520 con 0x7fdc580b2b10 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc56d9d640 1 -- 172.21.15.150:0/1388094380 >> v1:172.21.15.187:6789/0 conn(0x7fdc5815e010 legacy=0x7fdc580b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1115537549 0 0) 0x7fdc580d24a0 con 0x7fdc5815e010 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc56d9d640 1 -- 172.21.15.150:0/1388094380 >> v1:172.21.15.150:6789/0 conn(0x7fdc580b2b10 legacy=0x7fdc580cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.050+0000 7fdc56d9d640 1 -- 172.21.15.150:0/1388094380 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc580d2780 con 0x7fdc580b5ad0 2024-04-25T14:03:07.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.051+0000 7fdc56d9d640 1 --2- 172.21.15.150:0/1388094380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc580b5ad0 0x7fdc580d3260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc500097a0 tx=0x7fdc5002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dcf21a3bc181ca4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.051+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc50037070 con 0x7fdc580b5ad0 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.051+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc50032640 con 0x7fdc580b5ad0 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.051+0000 7fdc5559a640 1 -- 172.21.15.150:0/1388094380 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc500329f0 con 0x7fdc580b5ad0 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 -- 172.21.15.150:0/1388094380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc580b5ad0 msgr2=0x7fdc580d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/1388094380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc580b5ad0 0x7fdc580d3260 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdc500097a0 tx=0x7fdc5002a2c0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 -- 172.21.15.150:0/1388094380 shutdown_connections 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/1388094380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc580b5ad0 0x7fdc580d3260 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 -- 172.21.15.150:0/1388094380 >> 172.21.15.150:0/1388094380 conn(0x7fdc580d03c0 msgr2=0x7fdc580d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 -- 172.21.15.150:0/1388094380 shutdown_connections 2024-04-25T14:03:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.052+0000 7fdc5ce30640 1 -- 172.21.15.150:0/1388094380 wait complete. 2024-04-25T14:03:07.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.054+0000 7fdc5ce30640 1 Processor -- start 2024-04-25T14:03:07.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.054+0000 7fdc5ce30640 1 -- start start 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc580b2b10 0x7fdc580c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc58159bb0 0x7fdc58142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdc5815e420 con 0x7fdc5815e010 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdc580d2dd0 con 0x7fdc58159bb0 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc5ce30640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdc580b6d30 con 0x7fdc580b2b10 2024-04-25T14:03:07.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc55d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc55d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40752/0 (socket says 172.21.15.150:40752) 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.055+0000 7fdc55d9b640 1 -- 172.21.15.150:0/937520470 learned_addr learned my addr 172.21.15.150:0/937520470 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc56d9d640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc58159bb0 0x7fdc58142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc5659c640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc580b2b10 0x7fdc580c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 -- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc580b2b10 msgr2=0x7fdc580c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc580b2b10 0x7fdc580c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 -- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc58159bb0 msgr2=0x7fdc58142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc58159bb0 0x7fdc58142450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 -- 172.21.15.150:0/937520470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc58161280 con 0x7fdc5815e010 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc55d9b640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c410 tx=0x7fdc4c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c017070 con 0x7fdc5815e010 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc4c0035f0 con 0x7fdc5815e010 2024-04-25T14:03:07.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc4c0039e0 con 0x7fdc5815e010 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc580c9dd0 con 0x7fdc5815e010 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.056+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc5815e8d0 con 0x7fdc5815e010 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.057+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc58069860 con 0x7fdc5815e010 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.058+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdc4c01a070 con 0x7fdc5815e010 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.058+0000 7fdc46ffd640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 0x7fdc380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.058+0000 7fdc5659c640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 0x7fdc380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.058+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fdc4c094940 con 0x7fdc5815e010 2024-04-25T14:03:07.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.059+0000 7fdc5659c640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 0x7fdc380769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc48005c00 tx=0x7fdc48009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.063+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdc4c05e680 con 0x7fdc5815e010 2024-04-25T14:03:07.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.236+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdc58076a40 con 0x7fdc5815e010 2024-04-25T14:03:07.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.237+0000 7fdc46ffd640 1 -- 172.21.15.150:0/937520470 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7fdc4c062980 con 0x7fdc5815e010 2024-04-25T14:03:07.237 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:07.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 msgr2=0x7fdc380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 0x7fdc380769f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdc48005c00 tx=0x7fdc48009040 comp rx=0 tx=0).stop 2024-04-25T14:03:07.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 msgr2=0x7fdc58159670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fdc4c00c410 tx=0x7fdc4c00c8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 shutdown_connections 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdc380744e0 0x7fdc380769f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdc58159bb0 0x7fdc58142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdc5815e010 0x7fdc58159670 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 --2- 172.21.15.150:0/937520470 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdc580b2b10 0x7fdc580c61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 >> 172.21.15.150:0/937520470 conn(0x7fdc580d03c0 msgr2=0x7fdc580b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 shutdown_connections 2024-04-25T14:03:07.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.239+0000 7fdc5ce30640 1 -- 172.21.15.150:0/937520470 wait complete. 2024-04-25T14:03:07.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:03:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.360+0000 7f0135a96640 1 Processor -- start 2024-04-25T14:03:07.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.360+0000 7f0135a96640 1 -- start start 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f0135a96640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0130070820 0x7f01301609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f0135a96640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01300cdca0 0x7f0130160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f0135a96640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01300cf300 con 0x7f01300d03a0 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f0135a96640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f01300cff10 con 0x7f01300cdca0 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f0135a96640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f01300cf5e0 con 0x7f0130070820 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f012ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01300cdca0 0x7f0130160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f012f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f01300d03a0 0x7f01300cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:35044/0 (socket says 172.21.15.150:35044) 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f012f7fe640 1 -- 172.21.15.150:0/401907001 learned_addr learned my addr 172.21.15.150:0/401907001 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.361+0000 7f012effd640 1 --2- 172.21.15.150:0/401907001 >> v2:172.21.15.187:3300/0 conn(0x7f0130070820 0x7f01301609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.362+0000 7f012ffff640 1 -- 172.21.15.150:0/401907001 >> v2:172.21.15.187:3300/0 conn(0x7f0130070820 msgr2=0x7f01301609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:07.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.362+0000 7f012ffff640 1 --2- 172.21.15.150:0/401907001 >> v2:172.21.15.187:3300/0 conn(0x7f0130070820 0x7f01301609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.362+0000 7f012ffff640 1 -- 172.21.15.150:0/401907001 >> v1:172.21.15.150:6789/0 conn(0x7f01300d03a0 legacy=0x7f01300cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.362+0000 7f012ffff640 1 -- 172.21.15.150:0/401907001 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01300cf8c0 con 0x7f01300cdca0 2024-04-25T14:03:07.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.362+0000 7f012ffff640 1 --2- 172.21.15.150:0/401907001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300cdca0 0x7f0130160f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f01300715c0 tx=0x7f012802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700261f198d09f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f012e7fc640 1 -- 172.21.15.150:0/401907001 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128007740 con 0x7f01300cdca0 2024-04-25T14:03:07.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f012e7fc640 1 -- 172.21.15.150:0/401907001 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01280078e0 con 0x7f01300cdca0 2024-04-25T14:03:07.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f012e7fc640 1 -- 172.21.15.150:0/401907001 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128007cb0 con 0x7f01300cdca0 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 -- 172.21.15.150:0/401907001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300cdca0 msgr2=0x7f0130160f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 --2- 172.21.15.150:0/401907001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300cdca0 0x7f0130160f20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f01300715c0 tx=0x7f012802a680 comp rx=0 tx=0).stop 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 -- 172.21.15.150:0/401907001 shutdown_connections 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 --2- 172.21.15.150:0/401907001 >> v2:172.21.15.187:3300/0 conn(0x7f0130070820 0x7f01301609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 --2- 172.21.15.150:0/401907001 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300cdca0 0x7f0130160f20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.364+0000 7f0135a96640 1 -- 172.21.15.150:0/401907001 >> 172.21.15.150:0/401907001 conn(0x7f01300ce390 msgr2=0x7f01300ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.365+0000 7f0135a96640 1 -- 172.21.15.150:0/401907001 shutdown_connections 2024-04-25T14:03:07.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.365+0000 7f0135a96640 1 -- 172.21.15.150:0/401907001 wait complete. 2024-04-25T14:03:07.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.366+0000 7f0135a96640 1 Processor -- start 2024-04-25T14:03:07.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.366+0000 7f0135a96640 1 -- start start 2024-04-25T14:03:07.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0130070820 0x7f0130153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 0x7f01301e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f013006f190 con 0x7f01300d03a0 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f013006f010 con 0x7f0130070820 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f0135a96640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f013006f310 con 0x7f01300cdca0 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40770/0 (socket says 172.21.15.150:40770) 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 -- 172.21.15.150:0/3016291175 learned_addr learned my addr 172.21.15.150:0/3016291175 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012f7fe640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0130070820 0x7f0130153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012effd640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 0x7f01301e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 -- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 msgr2=0x7f01301e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 0x7f01301e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 -- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0130070820 msgr2=0x7f0130153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0130070820 0x7f0130153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 -- 172.21.15.150:0/3016291175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01300aeb30 con 0x7f01300d03a0 2024-04-25T14:03:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012effd640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 0x7f01301e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:07.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.368+0000 7f012ffff640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f012802ac70 tx=0x7f0128004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.371+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012803f070 con 0x7f01300d03a0 2024-04-25T14:03:07.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.371+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0128031bc0 con 0x7f01300d03a0 2024-04-25T14:03:07.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.371+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01300aee10 con 0x7f01300d03a0 2024-04-25T14:03:07.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.371+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0128044460 con 0x7f01300d03a0 2024-04-25T14:03:07.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.371+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01300af3d0 con 0x7f01300d03a0 2024-04-25T14:03:07.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.372+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0130076a40 con 0x7f01300d03a0 2024-04-25T14:03:07.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.372+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0128007790 con 0x7f01300d03a0 2024-04-25T14:03:07.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.373+0000 7f0117fff640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 0x7f011c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.373+0000 7f012f7fe640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 0x7f011c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.373+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f01280b6e60 con 0x7f01300d03a0 2024-04-25T14:03:07.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.374+0000 7f012f7fe640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 0x7f011c0769c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0120007fd0 tx=0x7f0120007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.378+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0128081280 con 0x7f01300d03a0 2024-04-25T14:03:07.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.552+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f013015e050 con 0x7f01300d03a0 2024-04-25T14:03:07.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.553+0000 7f0117fff640 1 -- 172.21.15.150:0/3016291175 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f0128080b80 con 0x7f01300d03a0 2024-04-25T14:03:07.553 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 msgr2=0x7f011c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 0x7f011c0769c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0120007fd0 tx=0x7f0120007e80 comp rx=0 tx=0).stop 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 msgr2=0x7f01301e8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f012802ac70 tx=0x7f0128004a40 comp rx=0 tx=0).stop 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 shutdown_connections 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f011c0744b0 0x7f011c0769c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01300d03a0 0x7f01301e8f70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01300cdca0 0x7f01301e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 --2- 172.21.15.150:0/3016291175 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0130070820 0x7f0130153440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 >> 172.21.15.150:0/3016291175 conn(0x7f01300ce390 msgr2=0x7f01300d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 shutdown_connections 2024-04-25T14:03:07.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.555+0000 7f0135a96640 1 -- 172.21.15.150:0/3016291175 wait complete. 2024-04-25T14:03:07.566 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:03:07.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.677+0000 7f91c8bf8640 1 Processor -- start 2024-04-25T14:03:07.677 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.677+0000 7f91c8bf8640 1 -- start start 2024-04-25T14:03:07.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c8bf8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c40b4c10 0x7f91c4160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c8bf8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f91c406d800 0x7f91c40d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c8bf8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91c40b9600 con 0x7f91c40b91f0 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c8bf8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f91c406cc60 con 0x7f91c406d800 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c8bf8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91c40d4f20 con 0x7f91c40b4c10 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c2d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f91c406d800 0x7f91c40d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c1d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f91c40b91f0 0x7f91c406d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46208/0 (socket says 172.21.15.150:46208) 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c1d9b640 1 -- 172.21.15.150:0/89036124 learned_addr learned my addr 172.21.15.150:0/89036124 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c259c640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c40b4c10 0x7f91c4160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c2d9d640 1 -- 172.21.15.150:0/89036124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c40b4c10 msgr2=0x7f91c4160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c2d9d640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c40b4c10 0x7f91c4160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c2d9d640 1 -- 172.21.15.150:0/89036124 >> v1:172.21.15.150:6789/0 conn(0x7f91c40b91f0 legacy=0x7f91c406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c159a640 1 -- 172.21.15.150:0/89036124 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1362016334 0 0) 0x7f91c40b9600 con 0x7f91c40b91f0 2024-04-25T14:03:07.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.679+0000 7f91c2d9d640 1 -- 172.21.15.150:0/89036124 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c40d5200 con 0x7f91c406d800 2024-04-25T14:03:07.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.680+0000 7f91c2d9d640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c406d800 0x7f91c40d0950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f91c406cde0 tx=0x7f91bc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ed0db6028b7a6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.680+0000 7f91c159a640 1 -- 172.21.15.150:0/89036124 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc037070 con 0x7f91c406d800 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.680+0000 7f91c159a640 1 -- 172.21.15.150:0/89036124 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91bc032440 con 0x7f91c406d800 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.680+0000 7f91c159a640 1 -- 172.21.15.150:0/89036124 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc032810 con 0x7f91c406d800 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 -- 172.21.15.150:0/89036124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c406d800 msgr2=0x7f91c40d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c406d800 0x7f91c40d0950 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f91c406cde0 tx=0x7f91bc02a180 comp rx=0 tx=0).stop 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 -- 172.21.15.150:0/89036124 shutdown_connections 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c406d800 0x7f91c40d0950 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/89036124 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c40b4c10 0x7f91c4160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 -- 172.21.15.150:0/89036124 >> 172.21.15.150:0/89036124 conn(0x7f91c40b7690 msgr2=0x7f91c40b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 -- 172.21.15.150:0/89036124 shutdown_connections 2024-04-25T14:03:07.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.681+0000 7f91c8bf8640 1 -- 172.21.15.150:0/89036124 wait complete. 2024-04-25T14:03:07.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.683+0000 7f91c8bf8640 1 Processor -- start 2024-04-25T14:03:07.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.683+0000 7f91c8bf8640 1 -- start start 2024-04-25T14:03:07.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c406d800 0x7f91c415b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c40b4c10 0x7f91c41e9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f91c406c8f0 con 0x7f91c40b91f0 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f91c406c610 con 0x7f91c40b4c10 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c8bf8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f91c406dc10 con 0x7f91c406d800 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c2d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59904/0 (socket says 172.21.15.150:59904) 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c2d9d640 1 -- 172.21.15.150:0/2923947235 learned_addr learned my addr 172.21.15.150:0/2923947235 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c259c640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c406d800 0x7f91c415b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.684+0000 7f91c1d9b640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c40b4c10 0x7f91c41e9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 -- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c406d800 msgr2=0x7f91c415b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c406d800 0x7f91c415b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 -- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c40b4c10 msgr2=0x7f91c41e9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c40b4c10 0x7f91c41e9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 -- 172.21.15.150:0/2923947235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91c41483d0 con 0x7f91c40b91f0 2024-04-25T14:03:07.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c2d9d640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009890 tx=0x7f91bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc037050 con 0x7f91c40b91f0 2024-04-25T14:03:07.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91bc038040 con 0x7f91c40b91f0 2024-04-25T14:03:07.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91c41486b0 con 0x7f91c40b91f0 2024-04-25T14:03:07.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91bc006e10 con 0x7f91c40b91f0 2024-04-25T14:03:07.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.685+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91c4148c70 con 0x7f91c40b91f0 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.686+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91c41ea350 con 0x7f91c40b91f0 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.687+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f91bc030980 con 0x7f91c40b91f0 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.687+0000 7f91b2ffd640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 0x7f91a40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.687+0000 7f91c259c640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 0x7f91a40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.687+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f91bc0b5f50 con 0x7f91c40b91f0 2024-04-25T14:03:07.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.688+0000 7f91c259c640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 0x7f91a40769f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91b4003aa0 tx=0x7f91b4009830 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.692+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91bc032e20 con 0x7f91c40b91f0 2024-04-25T14:03:07.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.867+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f91c40b7d80 con 0x7f91c40b91f0 2024-04-25T14:03:07.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.867+0000 7f91b2ffd640 1 -- 172.21.15.150:0/2923947235 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f91bc07fc90 con 0x7f91c40b91f0 2024-04-25T14:03:07.868 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:07.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 msgr2=0x7f91a40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 0x7f91a40769f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91b4003aa0 tx=0x7f91b4009830 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 msgr2=0x7f91c4149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f91bc009890 tx=0x7f91bc0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 shutdown_connections 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f91a40744e0 0x7f91a40769f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f91c40b91f0 0x7f91c4149470 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f91c40b4c10 0x7f91c41e9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 --2- 172.21.15.150:0/2923947235 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f91c406d800 0x7f91c415b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 >> 172.21.15.150:0/2923947235 conn(0x7f91c40b7690 msgr2=0x7f91c40bb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 shutdown_connections 2024-04-25T14:03:07.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.869+0000 7f91c8bf8640 1 -- 172.21.15.150:0/2923947235 wait complete. 2024-04-25T14:03:07.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:03:07.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.989+0000 7f92fd496640 1 Processor -- start 2024-04-25T14:03:07.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.989+0000 7f92fd496640 1 -- start start 2024-04-25T14:03:07.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92fd496640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f92f8160460 0x7f92f806ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92fd496640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92f8163d80 0x7f92f80b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92fd496640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f806ff50 con 0x7f92f80b5900 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92fd496640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f92f80b5d10 con 0x7f92f8163d80 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92fd496640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f92f806f3d0 con 0x7f92f8160460 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f92f8163d80 0x7f92f80b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f92f80b5900 0x7f92f8072f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46210/0 (socket says 172.21.15.150:46210) 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f77fe640 1 -- 172.21.15.150:0/699713848 learned_addr learned my addr 172.21.15.150:0/699713848 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/699713848 >> v2:172.21.15.187:3300/0 conn(0x7f92f8160460 0x7f92f806ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f67fc640 1 -- 172.21.15.150:0/699713848 >> v2:172.21.15.187:3300/0 conn(0x7f92f8160460 msgr2=0x7f92f806ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f67fc640 1 --2- 172.21.15.150:0/699713848 >> v2:172.21.15.187:3300/0 conn(0x7f92f8160460 0x7f92f806ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f67fc640 1 -- 172.21.15.150:0/699713848 >> v1:172.21.15.150:6789/0 conn(0x7f92f80b5900 legacy=0x7f92f8072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f67fc640 1 -- 172.21.15.150:0/699713848 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f806f6b0 con 0x7f92f8163d80 2024-04-25T14:03:07.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.991+0000 7f92f5ffb640 1 -- 172.21.15.150:0/699713848 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2170958072 0 0) 0x7f92f806ff50 con 0x7f92f80b5900 2024-04-25T14:03:07.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.992+0000 7f92f67fc640 1 --2- 172.21.15.150:0/699713848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b53c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92f80d13d0 tx=0x7f92ec02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e46e882697b541d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.992+0000 7f92f5ffb640 1 -- 172.21.15.150:0/699713848 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec037070 con 0x7f92f8163d80 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.992+0000 7f92f5ffb640 1 -- 172.21.15.150:0/699713848 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92ec032440 con 0x7f92f8163d80 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.992+0000 7f92f5ffb640 1 -- 172.21.15.150:0/699713848 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec02ae60 con 0x7f92f8163d80 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 -- 172.21.15.150:0/699713848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 msgr2=0x7f92f80b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 --2- 172.21.15.150:0/699713848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b53c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f92f80d13d0 tx=0x7f92ec02a110 comp rx=0 tx=0).stop 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 -- 172.21.15.150:0/699713848 shutdown_connections 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 --2- 172.21.15.150:0/699713848 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b53c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 --2- 172.21.15.150:0/699713848 >> v2:172.21.15.187:3300/0 conn(0x7f92f8160460 0x7f92f806ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 -- 172.21.15.150:0/699713848 >> 172.21.15.150:0/699713848 conn(0x7f92f80d03c0 msgr2=0x7f92f80d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 -- 172.21.15.150:0/699713848 shutdown_connections 2024-04-25T14:03:07.994 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.993+0000 7f92fd496640 1 -- 172.21.15.150:0/699713848 wait complete. 2024-04-25T14:03:07.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.995+0000 7f92fd496640 1 Processor -- start 2024-04-25T14:03:07.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.995+0000 7f92fd496640 1 -- start start 2024-04-25T14:03:07.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92f80b5900 0x7f92f81f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f81597d0 0x7f92f814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f92f80b3030 con 0x7f92f8163d80 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f92f806fc70 con 0x7f92f81597d0 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92fd496640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f92f80ce5c0 con 0x7f92f80b5900 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92f67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92f67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59936/0 (socket says 172.21.15.150:59936) 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.996+0000 7f92f67fc640 1 -- 172.21.15.150:0/13557679 learned_addr learned my addr 172.21.15.150:0/13557679 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92f80b5900 0x7f92f81f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f77fe640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f81597d0 0x7f92f814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 -- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92f80b5900 msgr2=0x7f92f81f6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.998 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92f80b5900 0x7f92f81f6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 -- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f81597d0 msgr2=0x7f92f814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f81597d0 0x7f92f814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 -- 172.21.15.150:0/13557679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f80b6600 con 0x7f92f8163d80 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92f67fc640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92ec000c00 tx=0x7f92ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec037040 con 0x7f92f8163d80 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92ec042070 con 0x7f92f8163d80 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f80b6870 con 0x7f92f8163d80 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ec030bd0 con 0x7f92f8163d80 2024-04-25T14:03:07.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.997+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f81f6a40 con 0x7f92f8163d80 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.998+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f8076a40 con 0x7f92f8163d80 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.999+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f92ec047460 con 0x7f92f8163d80 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.999+0000 7f92df7fe640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 0x7f92e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.999+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f92ec0b6150 con 0x7f92f8163d80 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:07.999+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 0x7f92e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.000+0000 7f92f6ffd640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 0x7f92e40769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92f806e7e0 tx=0x7f92e8003990 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.004+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f92ec032e20 con 0x7f92f8163d80 2024-04-25T14:03:08.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.190+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f92f8162f30 con 0x7f92f8163d80 2024-04-25T14:03:08.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.191+0000 7f92df7fe640 1 -- 172.21.15.150:0/13557679 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) ==== 80+0+0 (secure 0 0 0) 0x7f92ec080570 con 0x7f92f8163d80 2024-04-25T14:03:08.191 INFO:teuthology.orchestra.run.smithi150.stderr:listed 0 entries 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.192+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 msgr2=0x7f92e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.192+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 0x7f92e40769c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f92f806e7e0 tx=0x7f92e8003990 comp rx=0 tx=0).stop 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.192+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 msgr2=0x7f92f80b7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.192+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f92ec000c00 tx=0x7f92ec0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 shutdown_connections 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f92e40744b0 0x7f92e40769c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f92f81597d0 0x7f92f814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f92f8163d80 0x7f92f80b7350 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 --2- 172.21.15.150:0/13557679 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f92f80b5900 0x7f92f81f6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 >> 172.21.15.150:0/13557679 conn(0x7f92f80d03c0 msgr2=0x7f92f80d38c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:08.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 shutdown_connections 2024-04-25T14:03:08.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.193+0000 7f92fd496640 1 -- 172.21.15.150:0/13557679 wait complete. 2024-04-25T14:03:08.204 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-04-25T14:03:08.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:08.204 DEBUG:teuthology.orchestra.run.smithi187:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-04-25T14:03:08.235 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:03:08.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:08.235 DEBUG:teuthology.orchestra.run.smithi187:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-25T14:03:08.292 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T14:03:08.293 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:08.323 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:03:08.337 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:03:08.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.403+0000 7f4a37a92640 1 Processor -- start 2024-04-25T14:03:08.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.403+0000 7f4a37a92640 1 -- start start 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a37a92640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a3006e4d0 0x7f4a300b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a37a92640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4a300b3fe0 0x7f4a30167da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a37a92640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a300d2ee0 con 0x7f4a300b4940 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a37a92640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4a300d1030 con 0x7f4a3006e4d0 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a37a92640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4a30161b60 con 0x7f4a300b3fe0 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.404+0000 7f4a3502d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a3006e4d0 0x7f4a300b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4a3006e4d0 0x7f4a300b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59946/0 (socket says 172.21.15.150:59946) 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 -- 172.21.15.150:0/951140985 learned_addr learned my addr 172.21.15.150:0/951140985 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:08.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3602f640 1 --2- 172.21.15.150:0/951140985 >> v2:172.21.15.187:3300/0 conn(0x7f4a300b3fe0 0x7f4a30167da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 -- 172.21.15.150:0/951140985 >> v2:172.21.15.187:3300/0 conn(0x7f4a300b3fe0 msgr2=0x7f4a30167da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:08.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 --2- 172.21.15.150:0/951140985 >> v2:172.21.15.187:3300/0 conn(0x7f4a300b3fe0 0x7f4a30167da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 -- 172.21.15.150:0/951140985 >> v1:172.21.15.187:6789/0 conn(0x7f4a300b4940 legacy=0x7f4a300d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:08.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 -- 172.21.15.150:0/951140985 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a300d31c0 con 0x7f4a3006e4d0 2024-04-25T14:03:08.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.405+0000 7f4a3502d640 1 --2- 172.21.15.150:0/951140985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300b3aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4a300b57d0 tx=0x7f4a2002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f352d6fbb9fd40a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a3482c640 1 -- 172.21.15.150:0/951140985 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20037070 con 0x7f4a3006e4d0 2024-04-25T14:03:08.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a3482c640 1 -- 172.21.15.150:0/951140985 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a20032440 con 0x7f4a3006e4d0 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a3482c640 1 -- 172.21.15.150:0/951140985 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a20032810 con 0x7f4a3006e4d0 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 -- 172.21.15.150:0/951140985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 msgr2=0x7f4a300b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 --2- 172.21.15.150:0/951140985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300b3aa0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f4a300b57d0 tx=0x7f4a2002a1a0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 -- 172.21.15.150:0/951140985 shutdown_connections 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 --2- 172.21.15.150:0/951140985 >> v2:172.21.15.187:3300/0 conn(0x7f4a300b3fe0 0x7f4a30167da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 --2- 172.21.15.150:0/951140985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300b3aa0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.406+0000 7f4a37a92640 1 -- 172.21.15.150:0/951140985 >> 172.21.15.150:0/951140985 conn(0x7f4a3009b4a0 msgr2=0x7f4a3009b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.407+0000 7f4a37a92640 1 -- 172.21.15.150:0/951140985 shutdown_connections 2024-04-25T14:03:08.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.407+0000 7f4a37a92640 1 -- 172.21.15.150:0/951140985 wait complete. 2024-04-25T14:03:08.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.408+0000 7f4a37a92640 1 Processor -- start 2024-04-25T14:03:08.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.408+0000 7f4a37a92640 1 -- start start 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a300b3fe0 0x7f4a300cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a300b4940 0x7f4a301f4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4a300b43f0 con 0x7f4a3006e4d0 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3006cfa0 con 0x7f4a300b3fe0 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4a300b4570 con 0x7f4a300b4940 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59954/0 (socket says 172.21.15.150:59954) 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 -- 172.21.15.150:0/3240978383 learned_addr learned my addr 172.21.15.150:0/3240978383 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3502d640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a300b3fe0 0x7f4a300cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3602f640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a300b4940 0x7f4a301f4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 -- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a300b4940 msgr2=0x7f4a301f4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a300b4940 0x7f4a301f4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 -- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a300b3fe0 msgr2=0x7f4a300cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a300b3fe0 0x7f4a300cb010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 -- 172.21.15.150:0/3240978383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a301f4530 con 0x7f4a3006e4d0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a3582e640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00b3f0 tx=0x7f4a2c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c015020 con 0x7f4a3006e4d0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a2c011830 con 0x7f4a3006e4d0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a301f7f90 con 0x7f4a3006e4d0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c011c00 con 0x7f4a3006e4d0 2024-04-25T14:03:08.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.410+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a301f8550 con 0x7f4a3006e4d0 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.411+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a30076a40 con 0x7f4a3006e4d0 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.412+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4a2c018020 con 0x7f4a3006e4d0 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.413+0000 7f4a267fc640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 0x7f4a140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.413+0000 7f4a3502d640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 0x7f4a140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.413+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4a2c095010 con 0x7f4a3006e4d0 2024-04-25T14:03:08.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.413+0000 7f4a3502d640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 0x7f4a140769f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4a20034000 tx=0x7f4a2002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.417+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a2c01f070 con 0x7f4a3006e4d0 2024-04-25T14:03:08.626 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.626+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a300cab60 con 0x7f4a3006e4d0 2024-04-25T14:03:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.626+0000 7f4a267fc640 1 -- 172.21.15.150:0/3240978383 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f4a2c05f3a0 con 0x7f4a3006e4d0 2024-04-25T14:03:08.627 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:08.627 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:03:08.627 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 msgr2=0x7f4a140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 0x7f4a140769f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4a20034000 tx=0x7f4a2002e000 comp rx=0 tx=0).stop 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 msgr2=0x7f4a300c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00b3f0 tx=0x7f4a2c00b8b0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 shutdown_connections 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4a140744e0 0x7f4a140769f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4a300b4940 0x7f4a301f4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.628+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4a300b3fe0 0x7f4a300cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.629+0000 7f4a37a92640 1 --2- 172.21.15.150:0/3240978383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4a3006e4d0 0x7f4a300c63c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.629+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 >> 172.21.15.150:0/3240978383 conn(0x7f4a3009b4a0 msgr2=0x7f4a300d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.629+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 shutdown_connections 2024-04-25T14:03:08.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.629+0000 7f4a37a92640 1 -- 172.21.15.150:0/3240978383 wait complete. 2024-04-25T14:03:08.640 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.787+0000 7fbd8cba8640 1 Processor -- start 2024-04-25T14:03:08.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.787+0000 7fbd8cba8640 1 -- start start 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd8cba8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd8806d8f0 0x7fbd88154950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd8cba8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd88071600 con 0x7fbd88161b10 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd8cba8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd880aec30 con 0x7fbd880d1120 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd8cba8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbd880d1530 con 0x7fbd8806d8f0 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd86d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd8806d8f0 0x7fbd88154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd85d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbd88161b10 0x7fbd88161f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46216/0 (socket says 172.21.15.150:46216) 2024-04-25T14:03:08.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd85d9b640 1 -- 172.21.15.150:0/3570472864 learned_addr learned my addr 172.21.15.150:0/3570472864 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:08.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd86d9d640 1 -- 172.21.15.150:0/3570472864 >> v1:172.21.15.187:6789/0 conn(0x7fbd880d1120 legacy=0x7fbd88070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd86d9d640 1 -- 172.21.15.150:0/3570472864 >> v1:172.21.15.150:6789/0 conn(0x7fbd88161b10 legacy=0x7fbd88161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.789+0000 7fbd86d9d640 1 -- 172.21.15.150:0/3570472864 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd880aef10 con 0x7fbd8806d8f0 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.790+0000 7fbd86d9d640 1 --2- 172.21.15.150:0/3570472864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8806d8f0 0x7fbd88154950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009d90 tx=0x7fbd8002a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c25e54f274c666b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.790+0000 7fbd8559a640 1 -- 172.21.15.150:0/3570472864 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80031600 con 0x7fbd8806d8f0 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.790+0000 7fbd8559a640 1 -- 172.21.15.150:0/3570472864 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd800317a0 con 0x7fbd8806d8f0 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.790+0000 7fbd8559a640 1 -- 172.21.15.150:0/3570472864 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80031b70 con 0x7fbd8806d8f0 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3570472864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8806d8f0 msgr2=0x7fbd88154950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3570472864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8806d8f0 0x7fbd88154950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009d90 tx=0x7fbd8002a5d0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3570472864 shutdown_connections 2024-04-25T14:03:08.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3570472864 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8806d8f0 0x7fbd88154950 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3570472864 >> 172.21.15.150:0/3570472864 conn(0x7fbd880d1ae0 msgr2=0x7fbd880d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:08.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3570472864 shutdown_connections 2024-04-25T14:03:08.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.791+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3570472864 wait complete. 2024-04-25T14:03:08.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.793+0000 7fbd8cba8640 1 Processor -- start 2024-04-25T14:03:08.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.793+0000 7fbd8cba8640 1 -- start start 2024-04-25T14:03:08.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 0x7fbd880d0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 0x7fbd88146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbd880ae7e0 con 0x7fbd880d1120 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbd880af4d0 con 0x7fbd8806d8f0 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8cba8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd880ae960 con 0x7fbd88161b10 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59982/0 (socket says 172.21.15.150:59982) 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 -- 172.21.15.150:0/3468591842 learned_addr learned my addr 172.21.15.150:0/3468591842 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd86d9d640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 0x7fbd88146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8659c640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 0x7fbd880d0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 -- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 msgr2=0x7fbd88146ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 0x7fbd88146ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 -- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 msgr2=0x7fbd880d0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 0x7fbd880d0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd85d9b640 1 -- 172.21.15.150:0/3468591842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd88159330 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd86d9d640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 0x7fbd88146ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.794+0000 7fbd8659c640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 0x7fbd880d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd85d9b640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00c480 tx=0x7fbd7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c017070 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd7c005ce0 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd88159610 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd7c0072f0 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.795+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd88159bd0 con 0x7fbd880d1120 2024-04-25T14:03:08.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.796+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd88076a40 con 0x7fbd880d1120 2024-04-25T14:03:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.796+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbd7c01a070 con 0x7fbd880d1120 2024-04-25T14:03:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.797+0000 7fbd76ffd640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 0x7fbd680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.797+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fbd7c094800 con 0x7fbd880d1120 2024-04-25T14:03:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.797+0000 7fbd8659c640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 0x7fbd680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:08.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.797+0000 7fbd8659c640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 0x7fbd680769f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbd78001660 tx=0x7fbd78009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:08.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.801+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd7c05e660 con 0x7fbd880d1120 2024-04-25T14:03:08.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.977+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd880b4340 con 0x7fbd880d1120 2024-04-25T14:03:08.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.978+0000 7fbd76ffd640 1 -- 172.21.15.150:0/3468591842 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fbd7c062960 con 0x7fbd880d1120 2024-04-25T14:03:08.979 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:08.979 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 msgr2=0x7fbd680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 0x7fbd680769f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbd78001660 tx=0x7fbd78009040 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 msgr2=0x7fbd881f4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c00c480 tx=0x7fbd7c00c940 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 shutdown_connections 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd680744e0 0x7fbd680769f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd88161b10 0x7fbd88146ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd880d1120 0x7fbd881f4530 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 --2- 172.21.15.150:0/3468591842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8806d8f0 0x7fbd880d0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 >> 172.21.15.150:0/3468591842 conn(0x7fbd880d1ae0 msgr2=0x7fbd880b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 shutdown_connections 2024-04-25T14:03:08.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:08.980+0000 7fbd8cba8640 1 -- 172.21.15.150:0/3468591842 wait complete. 2024-04-25T14:03:08.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:02:06.806760+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:02:07.444080+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-25T14:02:11.170027+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '24', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-25T14:03:08.992 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:09.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.102+0000 7f30d944c640 1 Processor -- start 2024-04-25T14:03:09.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.102+0000 7f30d944c640 1 -- start start 2024-04-25T14:03:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d944c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30d40b4c10 0x7f30d4160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d944c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 0x7f30d406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d944c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30d40b9600 con 0x7f30d406d630 2024-04-25T14:03:09.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d944c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f30d406cc60 con 0x7f30d40b4c10 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d944c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f30d40d4f20 con 0x7f30d40b91f0 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f30d40b4c10 0x7f30d4160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f30d406d630 0x7f30d4163c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46230/0 (socket says 172.21.15.150:46230) 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d37fe640 1 -- 172.21.15.150:0/2052909674 learned_addr learned my addr 172.21.15.150:0/2052909674 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/2052909674 >> v2:172.21.15.150:3300/0 conn(0x7f30d40b4c10 0x7f30d4160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59990/0 (socket says 172.21.15.150:59990) 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d27fc640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 0x7f30d406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 -- 172.21.15.150:0/2052909674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 msgr2=0x7f30d406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 0x7f30d406d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 -- 172.21.15.150:0/2052909674 >> v1:172.21.15.150:6789/0 conn(0x7f30d406d630 legacy=0x7f30d4163c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d1ffb640 1 -- 172.21.15.150:0/2052909674 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3746773390 0 0) 0x7f30d40b9600 con 0x7f30d406d630 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d2ffd640 1 -- 172.21.15.150:0/2052909674 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d40d5200 con 0x7f30d40b4c10 2024-04-25T14:03:09.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.104+0000 7f30d27fc640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 0x7f30d406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:09.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.105+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b4c10 0x7f30d4160150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009d90 tx=0x7f30c402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da31779e6298d80 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.105+0000 7f30d1ffb640 1 -- 172.21.15.150:0/2052909674 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4007740 con 0x7f30d40b4c10 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.105+0000 7f30d1ffb640 1 -- 172.21.15.150:0/2052909674 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30c40078e0 con 0x7f30d40b4c10 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.105+0000 7f30d1ffb640 1 -- 172.21.15.150:0/2052909674 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30c4007cb0 con 0x7f30d40b4c10 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 -- 172.21.15.150:0/2052909674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b4c10 msgr2=0x7f30d4160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b4c10 0x7f30d4160150 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f30c4009d90 tx=0x7f30c402a660 comp rx=0 tx=0).stop 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 -- 172.21.15.150:0/2052909674 shutdown_connections 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d40b91f0 0x7f30d406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 --2- 172.21.15.150:0/2052909674 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b4c10 0x7f30d4160150 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 -- 172.21.15.150:0/2052909674 >> 172.21.15.150:0/2052909674 conn(0x7f30d40b7690 msgr2=0x7f30d40b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 -- 172.21.15.150:0/2052909674 shutdown_connections 2024-04-25T14:03:09.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.106+0000 7f30d944c640 1 -- 172.21.15.150:0/2052909674 wait complete. 2024-04-25T14:03:09.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.108+0000 7f30d944c640 1 Processor -- start 2024-04-25T14:03:09.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.108+0000 7f30d944c640 1 -- start start 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d406d630 0x7f30d4163230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 0x7f30d40cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f30d406c8f0 con 0x7f30d40b91f0 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f30d406c610 con 0x7f30d40b4c10 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d944c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f30d406da40 con 0x7f30d406d630 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59992/0 (socket says 172.21.15.150:59992) 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 -- 172.21.15.150:0/796358614 learned_addr learned my addr 172.21.15.150:0/796358614 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d406d630 0x7f30d4163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d27fc640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 0x7f30d40cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 -- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d406d630 msgr2=0x7f30d4163230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d406d630 0x7f30d4163230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 -- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 msgr2=0x7f30d40cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 0x7f30d40cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 -- 172.21.15.150:0/796358614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30d4149f50 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d27fc640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 0x7f30d40cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.109+0000 7f30d37fe640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00c410 tx=0x7f30cc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.110+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc017070 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.110+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30cc005ce0 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.110+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30d414a230 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.110+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30cc0072f0 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.110+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30d414a7f0 con 0x7f30d40b91f0 2024-04-25T14:03:09.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.111+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d4076a40 con 0x7f30d40b91f0 2024-04-25T14:03:09.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.111+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f30cc01a070 con 0x7f30d40b91f0 2024-04-25T14:03:09.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.112+0000 7f30bb7fe640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 0x7f30c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.112+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f30cc094820 con 0x7f30d40b91f0 2024-04-25T14:03:09.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.112+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 0x7f30c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.112+0000 7f30d2ffd640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 0x7f30c00769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f30c4034000 tx=0x7f30c402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.116+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f30cc05e5f0 con 0x7f30d40b91f0 2024-04-25T14:03:09.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.296+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f30d415ed40 con 0x7f30d40b91f0 2024-04-25T14:03:09.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.297+0000 7f30bb7fe640 1 -- 172.21.15.150:0/796358614 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f30cc0628f0 con 0x7f30d40b91f0 2024-04-25T14:03:09.297 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 msgr2=0x7f30c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 0x7f30c00769c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f30c4034000 tx=0x7f30c402f000 comp rx=0 tx=0).stop 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 msgr2=0x7f30d40b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f30cc00c410 tx=0x7f30cc00c8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 shutdown_connections 2024-04-25T14:03:09.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f30c00744b0 0x7f30c00769c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f30d40b91f0 0x7f30d40b60c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f30d40b4c10 0x7f30d40cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 --2- 172.21.15.150:0/796358614 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f30d406d630 0x7f30d4163230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 >> 172.21.15.150:0/796358614 conn(0x7f30d40b7690 msgr2=0x7f30d40b5950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 shutdown_connections 2024-04-25T14:03:09.300 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.299+0000 7f30d944c640 1 -- 172.21.15.150:0/796358614 wait complete. 2024-04-25T14:03:09.310 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:03:09.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.421+0000 7f9fdce96640 1 Processor -- start 2024-04-25T14:03:09.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.421+0000 7f9fdce96640 1 -- start start 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fdce96640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 0x7f9fd815d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fdce96640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fd815ddd0 0x7f9fd806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fdce96640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fd80cbfd0 con 0x7f9fd815abd0 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fdce96640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9fd80d0fe0 con 0x7f9fd815ddd0 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fdce96640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd80cbd80 con 0x7f9fd815d480 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fd815ddd0 0x7f9fd806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9fd815ddd0 0x7f9fd806f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60006/0 (socket says 172.21.15.150:60006) 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 -- 172.21.15.150:0/62322131 learned_addr learned my addr 172.21.15.150:0/62322131 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd5d9b640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 0x7f9fd815d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 -- 172.21.15.150:0/62322131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 msgr2=0x7f9fd815d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 0x7f9fd815d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 -- 172.21.15.150:0/62322131 >> v1:172.21.15.187:6789/0 conn(0x7f9fd815abd0 legacy=0x7f9fd80cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:09.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd6d9d640 1 -- 172.21.15.150:0/62322131 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd80cc2b0 con 0x7f9fd815ddd0 2024-04-25T14:03:09.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.423+0000 7f9fd5d9b640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 0x7f9fd815d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.424+0000 7f9fd6d9d640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815ddd0 0x7f9fd806f010 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098a0 tx=0x7f9fd002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7941a0e3a24ef288 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.424+0000 7f9fd559a640 1 -- 172.21.15.150:0/62322131 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0037070 con 0x7f9fd815ddd0 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.424+0000 7f9fd559a640 1 -- 172.21.15.150:0/62322131 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fd0032440 con 0x7f9fd815ddd0 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.424+0000 7f9fd559a640 1 -- 172.21.15.150:0/62322131 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fd0032810 con 0x7f9fd815ddd0 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 -- 172.21.15.150:0/62322131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815ddd0 msgr2=0x7f9fd806f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815ddd0 0x7f9fd806f010 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f9fd00098a0 tx=0x7f9fd002a1a0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 -- 172.21.15.150:0/62322131 shutdown_connections 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815ddd0 0x7f9fd806f010 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 --2- 172.21.15.150:0/62322131 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815d480 0x7f9fd815d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 -- 172.21.15.150:0/62322131 >> 172.21.15.150:0/62322131 conn(0x7f9fd80ce390 msgr2=0x7f9fd80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 -- 172.21.15.150:0/62322131 shutdown_connections 2024-04-25T14:03:09.426 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.425+0000 7f9fdce96640 1 -- 172.21.15.150:0/62322131 wait complete. 2024-04-25T14:03:09.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.427+0000 7f9fdce96640 1 Processor -- start 2024-04-25T14:03:09.427 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.427+0000 7f9fdce96640 1 -- start start 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fd815abd0 0x7f9fd80b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815ddd0 0x7f9fd80b6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd80d0510 con 0x7f9fd815d480 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9fd80cfff0 con 0x7f9fd815abd0 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fdce96640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9fd815e1e0 con 0x7f9fd815ddd0 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fd5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fd5d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60008/0 (socket says 172.21.15.150:60008) 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.428+0000 7f9fd5d9b640 1 -- 172.21.15.150:0/3219809966 learned_addr learned my addr 172.21.15.150:0/3219809966 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd659c640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fd815abd0 0x7f9fd80b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd6d9d640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815ddd0 0x7f9fd80b6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 -- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815ddd0 msgr2=0x7f9fd80b6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.430 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815ddd0 0x7f9fd80b6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 -- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fd815abd0 msgr2=0x7f9fd80b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fd815abd0 0x7f9fd80b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 -- 172.21.15.150:0/3219809966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fd80b4d20 con 0x7f9fd815d480 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fd5d9b640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00eb40 tx=0x7f9fcc00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc007c10 con 0x7f9fd815d480 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fcc007db0 con 0x7f9fd815d480 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fd80b5000 con 0x7f9fd815d480 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fcc010680 con 0x7f9fd815d480 2024-04-25T14:03:09.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.429+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fd80b55c0 con 0x7f9fd815d480 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.430+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fd80b4a60 con 0x7f9fd815d480 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.431+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9fcc00b030 con 0x7f9fd815d480 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.431+0000 7f9fc6ffd640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 0x7f9fb80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.431+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f9fcc097910 con 0x7f9fd815d480 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.431+0000 7f9fd659c640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 0x7f9fb80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.432+0000 7f9fd659c640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 0x7f9fb80769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80cd070 tx=0x7f9fc800d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.436 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.436+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9fcc061650 con 0x7f9fd815d480 2024-04-25T14:03:09.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.611+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9fd8076a40 con 0x7f9fd815d480 2024-04-25T14:03:09.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.611+0000 7f9fc6ffd640 1 -- 172.21.15.150:0/3219809966 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) ==== 104+0+11 (secure 0 0 0) 0x7f9fcc065950 con 0x7f9fd815d480 2024-04-25T14:03:09.611 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:09.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 msgr2=0x7f9fb80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 0x7f9fb80769c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9fd80cd070 tx=0x7f9fc800d040 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 msgr2=0x7f9fd8153b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc00eb40 tx=0x7f9fcc00cf70 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 shutdown_connections 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9fb80744b0 0x7f9fb80769c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9fd815ddd0 0x7f9fd80b6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9fd815d480 0x7f9fd8153b60 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 --2- 172.21.15.150:0/3219809966 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9fd815abd0 0x7f9fd80b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 >> 172.21.15.150:0/3219809966 conn(0x7f9fd80ce390 msgr2=0x7f9fd815cbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 shutdown_connections 2024-04-25T14:03:09.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.613+0000 7f9fdce96640 1 -- 172.21.15.150:0/3219809966 wait complete. 2024-04-25T14:03:09.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:03:09.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.735+0000 7f5d42ab6640 1 Processor -- start 2024-04-25T14:03:09.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.735+0000 7f5d42ab6640 1 -- start start 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5d3c0ce4d0 0x7f5d3c0b4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0b32a0 0x7f5d3c163f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3c0cf5e0 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5d3c0ce9d0 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.736+0000 7f5d42ab6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3c0cecb0 con 0x7f5d3c0b32a0 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60024/0 (socket says 172.21.15.150:60024) 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 -- 172.21.15.150:0/3457282542 learned_addr learned my addr 172.21.15.150:0/3457282542 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/3457282542 >> v2:172.21.15.187:3300/0 conn(0x7f5d3c0ce4d0 0x7f5d3c0b4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d41053640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0b32a0 0x7f5d3c163f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 -- 172.21.15.150:0/3457282542 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0b32a0 msgr2=0x7f5d3c163f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0b32a0 0x7f5d3c163f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 -- 172.21.15.150:0/3457282542 >> v2:172.21.15.187:3300/0 conn(0x7f5d3c0ce4d0 msgr2=0x7f5d3c0b4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 --2- 172.21.15.150:0/3457282542 >> v2:172.21.15.187:3300/0 conn(0x7f5d3c0ce4d0 0x7f5d3c0b4090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.737+0000 7f5d40852640 1 -- 172.21.15.150:0/3457282542 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c0cef90 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d40852640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0724e0 tx=0x7f5d3002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80df113be64ebd6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/3457282542 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30037070 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/3457282542 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d30032440 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d3b7fe640 1 -- 172.21.15.150:0/3457282542 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d30032810 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 -- 172.21.15.150:0/3457282542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0d1bc0 msgr2=0x7f5d3c0b3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c0724e0 tx=0x7f5d3002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 -- 172.21.15.150:0/3457282542 shutdown_connections 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0b32a0 0x7f5d3c163f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/3457282542 >> v2:172.21.15.187:3300/0 conn(0x7f5d3c0ce4d0 0x7f5d3c0b4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/3457282542 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0d1bc0 0x7f5d3c0b3b50 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 -- 172.21.15.150:0/3457282542 >> 172.21.15.150:0/3457282542 conn(0x7f5d3c09b4a0 msgr2=0x7f5d3c0b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 -- 172.21.15.150:0/3457282542 shutdown_connections 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.738+0000 7f5d42ab6640 1 -- 172.21.15.150:0/3457282542 wait complete. 2024-04-25T14:03:09.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.740+0000 7f5d42ab6640 1 Processor -- start 2024-04-25T14:03:09.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.740+0000 7f5d42ab6640 1 -- start start 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d3c0b32a0 0x7f5d3c14bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0d1bc0 0x7f5d3c14e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c160660 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1604e0 con 0x7f5d3c0b32a0 2024-04-25T14:03:09.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5d3c1607e0 con 0x7f5d3c0d1bc0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60036/0 (socket says 172.21.15.150:60036) 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 -- 172.21.15.150:0/1131701409 learned_addr learned my addr 172.21.15.150:0/1131701409 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d41053640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0d1bc0 0x7f5d3c14e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d40852640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d3c0b32a0 0x7f5d3c14bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 -- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0d1bc0 msgr2=0x7f5d3c14e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0d1bc0 0x7f5d3c14e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 -- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d3c0b32a0 msgr2=0x7f5d3c14bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d3c0b32a0 0x7f5d3c14bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 -- 172.21.15.150:0/1131701409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c14c490 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d3bfff640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800e8a0 tx=0x7f5d2800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d28017070 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d2800c940 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3c1615a0 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2800cd30 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.742+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3c161b60 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.744+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d3c076a40 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.744+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5d2801a070 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.744+0000 7f5d397fa640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:09.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.744+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f5d280953d0 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.744+0000 7f5d40852640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:09.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.745+0000 7f5d40852640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 0x7f5d200769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d30034000 tx=0x7f5d3002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:09.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.749+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d2805f180 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.923 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.923+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5d3c1f7850 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.924+0000 7f5d397fa640 1 -- 172.21.15.150:0/1131701409 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) ==== 100+0+11 (secure 0 0 0) 0x7f5d28063480 con 0x7f5d3c0ce4d0 2024-04-25T14:03:09.924 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:03:09.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 msgr2=0x7f5d200769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 0x7f5d200769c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5d30034000 tx=0x7f5d3002e000 comp rx=0 tx=0).stop 2024-04-25T14:03:09.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 msgr2=0x7f5d3c1f6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:09.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800e8a0 tx=0x7f5d2800ed60 comp rx=0 tx=0).stop 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 shutdown_connections 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5d200744b0 0x7f5d200769c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5d3c0d1bc0 0x7f5d3c14e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5d3c0ce4d0 0x7f5d3c1f6d90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 --2- 172.21.15.150:0/1131701409 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5d3c0b32a0 0x7f5d3c14bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 >> 172.21.15.150:0/1131701409 conn(0x7f5d3c09b4a0 msgr2=0x7f5d3c0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 shutdown_connections 2024-04-25T14:03:09.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:09.926+0000 7f5d42ab6640 1 -- 172.21.15.150:0/1131701409 wait complete. 2024-04-25T14:03:09.937 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-25T14:03:10.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.047+0000 7fba98c84640 1 Processor -- start 2024-04-25T14:03:10.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.047+0000 7fba98c84640 1 -- start start 2024-04-25T14:03:10.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba98c84640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba941628f0 0x7fba940b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba98c84640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 0x7fba9415e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba98c84640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba940b37b0 con 0x7fba940b3a90 2024-04-25T14:03:10.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba98c84640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fba94161e10 con 0x7fba941628f0 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba98c84640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fba941620f0 con 0x7fba9406e130 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba92d9d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fba940b3a90 0x7fba940b3ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46234/0 (socket says 172.21.15.150:46234) 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9259c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fba941628f0 0x7fba940b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba92d9d640 1 -- 172.21.15.150:0/2708153591 learned_addr learned my addr 172.21.15.150:0/2708153591 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba91d9b640 1 --2- 172.21.15.150:0/2708153591 >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 0x7fba9415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9259c640 1 -- 172.21.15.150:0/2708153591 >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 msgr2=0x7fba9415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9259c640 1 --2- 172.21.15.150:0/2708153591 >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 0x7fba9415e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9259c640 1 -- 172.21.15.150:0/2708153591 >> v1:172.21.15.150:6789/0 conn(0x7fba940b3a90 legacy=0x7fba940b3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9259c640 1 -- 172.21.15.150:0/2708153591 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba941623d0 con 0x7fba941628f0 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba9159a640 1 -- 172.21.15.150:0/2708153591 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859445702 0 0) 0x7fba940b37b0 con 0x7fba940b3a90 2024-04-25T14:03:10.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.049+0000 7fba91d9b640 1 --2- 172.21.15.150:0/2708153591 >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 0x7fba9415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:10.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.050+0000 7fba9259c640 1 --2- 172.21.15.150:0/2708153591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940b31d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fba94073ad0 tx=0x7fba8402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4940a47f40dddb72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.050+0000 7fba9159a640 1 -- 172.21.15.150:0/2708153591 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84007740 con 0x7fba941628f0 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.050+0000 7fba9159a640 1 -- 172.21.15.150:0/2708153591 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba840078e0 con 0x7fba941628f0 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.050+0000 7fba9159a640 1 -- 172.21.15.150:0/2708153591 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84007cb0 con 0x7fba941628f0 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 -- 172.21.15.150:0/2708153591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 msgr2=0x7fba940b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 --2- 172.21.15.150:0/2708153591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940b31d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fba94073ad0 tx=0x7fba8402a680 comp rx=0 tx=0).stop 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 -- 172.21.15.150:0/2708153591 shutdown_connections 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 --2- 172.21.15.150:0/2708153591 >> v2:172.21.15.187:3300/0 conn(0x7fba9406e130 0x7fba9415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 --2- 172.21.15.150:0/2708153591 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940b31d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 -- 172.21.15.150:0/2708153591 >> 172.21.15.150:0/2708153591 conn(0x7fba9409b4a0 msgr2=0x7fba940b6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 -- 172.21.15.150:0/2708153591 shutdown_connections 2024-04-25T14:03:10.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.051+0000 7fba98c84640 1 -- 172.21.15.150:0/2708153591 wait complete. 2024-04-25T14:03:10.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.053+0000 7fba98c84640 1 Processor -- start 2024-04-25T14:03:10.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.053+0000 7fba98c84640 1 -- start start 2024-04-25T14:03:10.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba9406e130 0x7fba94150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba940b3a90 0x7fba94161310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fba940b2050 con 0x7fba941628f0 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fba940b1ed0 con 0x7fba9406e130 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba98c84640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fba940b21d0 con 0x7fba940b3a90 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60050/0 (socket says 172.21.15.150:60050) 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 -- 172.21.15.150:0/2892899713 learned_addr learned my addr 172.21.15.150:0/2892899713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba91d9b640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba940b3a90 0x7fba94161310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba9259c640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba9406e130 0x7fba94150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 -- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba940b3a90 msgr2=0x7fba94161310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba940b3a90 0x7fba94161310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 -- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba9406e130 msgr2=0x7fba94150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba9406e130 0x7fba94150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.054+0000 7fba92d9d640 1 -- 172.21.15.150:0/2892899713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba940c9910 con 0x7fba941628f0 2024-04-25T14:03:10.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba92d9d640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c440 tx=0x7fba8c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c0074e0 con 0x7fba941628f0 2024-04-25T14:03:10.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba8c0047e0 con 0x7fba941628f0 2024-04-25T14:03:10.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba940c9bf0 con 0x7fba941628f0 2024-04-25T14:03:10.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba8c004bd0 con 0x7fba941628f0 2024-04-25T14:03:10.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.055+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba940ca1b0 con 0x7fba941628f0 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.056+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba94076a40 con 0x7fba941628f0 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.056+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fba8c011070 con 0x7fba941628f0 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.057+0000 7fba82ffd640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 0x7fba740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.057+0000 7fba9259c640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 0x7fba740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.057+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fba8c095050 con 0x7fba941628f0 2024-04-25T14:03:10.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.057+0000 7fba9259c640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 0x7fba740769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fba84034000 tx=0x7fba8402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.061+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fba8c05ed90 con 0x7fba941628f0 2024-04-25T14:03:10.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.236+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fba940b2f90 con 0x7fba941628f0 2024-04-25T14:03:10.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.237+0000 7fba82ffd640 1 -- 172.21.15.150:0/2892899713 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7fba8c063090 con 0x7fba941628f0 2024-04-25T14:03:10.237 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 msgr2=0x7fba740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 0x7fba740769f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fba84034000 tx=0x7fba8402f000 comp rx=0 tx=0).stop 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 msgr2=0x7fba940c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00c440 tx=0x7fba8c00c900 comp rx=0 tx=0).stop 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 shutdown_connections 2024-04-25T14:03:10.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fba740744e0 0x7fba740769f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fba941628f0 0x7fba940c90c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fba940b3a90 0x7fba94161310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 --2- 172.21.15.150:0/2892899713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fba9406e130 0x7fba94150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 >> 172.21.15.150:0/2892899713 conn(0x7fba9409b4a0 msgr2=0x7fba9406fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 shutdown_connections 2024-04-25T14:03:10.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.239+0000 7fba98c84640 1 -- 172.21.15.150:0/2892899713 wait complete. 2024-04-25T14:03:10.250 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:10.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.358+0000 7f2d3e77a640 1 Processor -- start 2024-04-25T14:03:10.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.358+0000 7f2d3e77a640 1 -- start start 2024-04-25T14:03:10.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d3e77a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f2d38158760 0x7f2d381625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d3e77a640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3815f100 con 0x7f2d380b5400 2024-04-25T14:03:10.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d3e77a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d3815f3e0 con 0x7f2d3815e520 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d3e77a640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f2d38070fd0 con 0x7f2d38158760 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d377fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f2d380b5400 0x7f2d380b5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46244/0 (socket says 172.21.15.150:46244) 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.359+0000 7f2d377fe640 1 -- 172.21.15.150:0/2974757733 learned_addr learned my addr 172.21.15.150:0/2974757733 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d37fff640 1 --2- 172.21.15.150:0/2974757733 >> v2:172.21.15.187:3300/0 conn(0x7f2d38158760 0x7f2d381625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3083989050 0 0) 0x7f2d3815f100 con 0x7f2d380b5400 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d20003520 con 0x7f2d380b5400 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2195090305 0 0) 0x7f2d3815f3e0 con 0x7f2d3815e520 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d3815f100 con 0x7f2d3815e520 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 392093262 0 0) 0x7f2d20003520 con 0x7f2d380b5400 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d37fff640 1 -- 172.21.15.150:0/2974757733 >> v1:172.21.15.187:6789/0 conn(0x7f2d3815e520 legacy=0x7f2d3815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d37fff640 1 -- 172.21.15.150:0/2974757733 >> v1:172.21.15.150:6789/0 conn(0x7f2d380b5400 legacy=0x7f2d380b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.360+0000 7f2d37fff640 1 -- 172.21.15.150:0/2974757733 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38161b10 con 0x7f2d38158760 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.361+0000 7f2d37fff640 1 --2- 172.21.15.150:0/2974757733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d38158760 0x7f2d381625f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0097a0 tx=0x7f2d2c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d11be059b5e6f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.361+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c037070 con 0x7f2d38158760 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.361+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d2c02ac50 con 0x7f2d38158760 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.361+0000 7f2d36ffd640 1 -- 172.21.15.150:0/2974757733 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d28001670 con 0x7f2d38158760 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2974757733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d38158760 msgr2=0x7f2d381625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2974757733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d38158760 0x7f2d381625f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0097a0 tx=0x7f2d2c02a1c0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2974757733 shutdown_connections 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2974757733 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d38158760 0x7f2d381625f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2974757733 >> 172.21.15.150:0/2974757733 conn(0x7f2d380d0400 msgr2=0x7f2d380d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2974757733 shutdown_connections 2024-04-25T14:03:10.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.362+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2974757733 wait complete. 2024-04-25T14:03:10.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.363+0000 7f2d3e77a640 1 Processor -- start 2024-04-25T14:03:10.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.364+0000 7f2d3e77a640 1 -- start start 2024-04-25T14:03:10.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d380b5400 0x7f2d38151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d38158760 0x7f2d38155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2d380703d0 con 0x7f2d3815e520 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2d381620d0 con 0x7f2d380b5400 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3e77a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2d380ce370 con 0x7f2d38158760 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60060/0 (socket says 172.21.15.150:60060) 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 -- 172.21.15.150:0/2209364808 learned_addr learned my addr 172.21.15.150:0/2209364808 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d37fff640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d380b5400 0x7f2d38151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d377fe640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d38158760 0x7f2d38155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 -- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d38158760 msgr2=0x7f2d38155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d38158760 0x7f2d38155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 -- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d380b5400 msgr2=0x7f2d38151850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d380b5400 0x7f2d38151850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.365+0000 7f2d3cd17640 1 -- 172.21.15.150:0/2209364808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38159f80 con 0x7f2d3815e520 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d3cd17640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000c710 tx=0x7f2d3000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d30003b70 con 0x7f2d3815e520 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d30003d10 con 0x7f2d3815e520 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d3815a260 con 0x7f2d3815e520 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d3000aa50 con 0x7f2d3815e520 2024-04-25T14:03:10.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.366+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d3815a820 con 0x7f2d3815e520 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.367+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d38076a40 con 0x7f2d3815e520 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.367+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2d3000abf0 con 0x7f2d3815e520 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.368+0000 7f2d34ff9640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 0x7f2d240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.368+0000 7f2d37fff640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 0x7f2d240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.368+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f2d30095360 con 0x7f2d3815e520 2024-04-25T14:03:10.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.369+0000 7f2d37fff640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 0x7f2d240769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c034000 tx=0x7f2d2c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.373+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2d3000ce20 con 0x7f2d3815e520 2024-04-25T14:03:10.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.588+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d380d1c80 con 0x7f2d3815e520 2024-04-25T14:03:10.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.589+0000 7f2d34ff9640 1 -- 172.21.15.150:0/2209364808 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f2d3005f130 con 0x7f2d3815e520 2024-04-25T14:03:10.589 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:10.589 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:03:10.589 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:03:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 msgr2=0x7f2d240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 0x7f2d240769c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c034000 tx=0x7f2d2c02e000 comp rx=0 tx=0).stop 2024-04-25T14:03:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 msgr2=0x7f2d38153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f2d3000c710 tx=0x7f2d3000cbd0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 shutdown_connections 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f2d240744b0 0x7f2d240769c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2d3815e520 0x7f2d38153e40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2d38158760 0x7f2d38155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 --2- 172.21.15.150:0/2209364808 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2d380b5400 0x7f2d38151850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 >> 172.21.15.150:0/2209364808 conn(0x7f2d380d0400 msgr2=0x7f2d380d3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 shutdown_connections 2024-04-25T14:03:10.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.591+0000 7f2d3e77a640 1 -- 172.21.15.150:0/2209364808 wait complete. 2024-04-25T14:03:10.603 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:10.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.710+0000 7fa9a4f27640 1 Processor -- start 2024-04-25T14:03:10.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.710+0000 7fa9a4f27640 1 -- start start 2024-04-25T14:03:10.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa9a4f27640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9a00b5ab0 0x7fa9a0160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa9a4f27640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9a0164bc0 con 0x7fa9a00d2e90 2024-04-25T14:03:10.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa9a4f27640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9a00ce680 con 0x7fa9a00b2b10 2024-04-25T14:03:10.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa9a4f27640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa9a00b7db0 con 0x7fa9a00b5ab0 2024-04-25T14:03:10.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa9a00b5ab0 0x7fa9a0160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99dd9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa9a00d2e90 0x7fa9a00b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46250/0 (socket says 172.21.15.150:46250) 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99dd9b640 1 -- 172.21.15.150:0/1294721934 learned_addr learned my addr 172.21.15.150:0/1294721934 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99ed9d640 1 -- 172.21.15.150:0/1294721934 >> v1:172.21.15.187:6789/0 conn(0x7fa9a00b2b10 legacy=0x7fa9a00cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99ed9d640 1 -- 172.21.15.150:0/1294721934 >> v1:172.21.15.150:6789/0 conn(0x7fa9a00d2e90 legacy=0x7fa9a00b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99ed9d640 1 -- 172.21.15.150:0/1294721934 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a01648e0 con 0x7fa9a00b5ab0 2024-04-25T14:03:10.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.712+0000 7fa99d59a640 1 -- 172.21.15.150:0/1294721934 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351352424 0 0) 0x7fa9a0164bc0 con 0x7fa9a00d2e90 2024-04-25T14:03:10.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.713+0000 7fa99ed9d640 1 --2- 172.21.15.150:0/1294721934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00b5ab0 0x7fa9a0160220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097d0 tx=0x7fa99802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57c3ca548f5a3d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.713+0000 7fa99d59a640 1 -- 172.21.15.150:0/1294721934 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998037070 con 0x7fa9a00b5ab0 2024-04-25T14:03:10.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.713+0000 7fa99d59a640 1 -- 172.21.15.150:0/1294721934 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa99802ae40 con 0x7fa9a00b5ab0 2024-04-25T14:03:10.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.713+0000 7fa99d59a640 1 -- 172.21.15.150:0/1294721934 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa998032660 con 0x7fa9a00b5ab0 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1294721934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00b5ab0 msgr2=0x7fa9a0160220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1294721934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00b5ab0 0x7fa9a0160220 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa9980097d0 tx=0x7fa99802a130 comp rx=0 tx=0).stop 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1294721934 shutdown_connections 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1294721934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00b5ab0 0x7fa9a0160220 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1294721934 >> 172.21.15.150:0/1294721934 conn(0x7fa9a00d03a0 msgr2=0x7fa9a00d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1294721934 shutdown_connections 2024-04-25T14:03:10.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.714+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1294721934 wait complete. 2024-04-25T14:03:10.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.716+0000 7fa9a4f27640 1 Processor -- start 2024-04-25T14:03:10.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.716+0000 7fa9a4f27640 1 -- start start 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9a00d1c50 0x7fa9a00700e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 0x7fa9a0156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a00b8350 con 0x7fa9a00d2e90 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa9a0164690 con 0x7fa9a0070620 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa9a4f27640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa9a00b9720 con 0x7fa9a00d1c50 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60078/0 (socket says 172.21.15.150:60078) 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99e59c640 1 -- 172.21.15.150:0/1774316662 learned_addr learned my addr 172.21.15.150:0/1774316662 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99ed9d640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 0x7fa9a0156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99dd9b640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9a00d1c50 0x7fa9a00700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.717+0000 7fa99e59c640 1 -- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9a00d1c50 msgr2=0x7fa9a00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99e59c640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9a00d1c50 0x7fa9a00700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99e59c640 1 -- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 msgr2=0x7fa9a0156740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99e59c640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 0x7fa9a0156740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99e59c640 1 -- 172.21.15.150:0/1774316662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9a014a470 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99ed9d640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 0x7fa9a0156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa99e59c640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c760 tx=0x7fa99000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa990003b70 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa990003d30 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9a0070a30 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99000aa90 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.718+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9a0160dd0 con 0x7fa9a00d2e90 2024-04-25T14:03:10.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.719+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9a0076a40 con 0x7fa9a00d2e90 2024-04-25T14:03:10.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.719+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa99000ac30 con 0x7fa9a00d2e90 2024-04-25T14:03:10.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.720+0000 7fa98effd640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 0x7fa9800769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:10.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.720+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7fa9900955c0 con 0x7fa9a00d2e90 2024-04-25T14:03:10.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.720+0000 7fa99dd9b640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 0x7fa9800769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:10.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.720+0000 7fa99dd9b640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 0x7fa9800769f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0163810 tx=0x7fa994007d20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:10.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.725+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa9a0070a30 con 0x7fa9a00d2e90 2024-04-25T14:03:10.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.902+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9a00b3060 con 0x7fa9a00d2e90 2024-04-25T14:03:10.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.903+0000 7fa98effd640 1 -- 172.21.15.150:0/1774316662 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7fa9a014a470 con 0x7fa9a00d2e90 2024-04-25T14:03:10.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:10.903 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:10.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 msgr2=0x7fa9800769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 0x7fa9800769f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa9a0163810 tx=0x7fa994007d20 comp rx=0 tx=0).stop 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 msgr2=0x7fa9a0149f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa99000c760 tx=0x7fa99000cc20 comp rx=0 tx=0).stop 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 shutdown_connections 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa9800744e0 0x7fa9800769f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa9a0070620 0x7fa9a0156740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa9a00d1c50 0x7fa9a00700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 --2- 172.21.15.150:0/1774316662 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa9a00d2e90 0x7fa9a0149f30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:10.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 >> 172.21.15.150:0/1774316662 conn(0x7fa9a00d03a0 msgr2=0x7fa9a015e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:10.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 shutdown_connections 2024-04-25T14:03:10.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:10.905+0000 7fa9a4f27640 1 -- 172.21.15.150:0/1774316662 wait complete. 2024-04-25T14:03:10.917 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T14:03:10.917 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:11.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.024+0000 7f987e053640 1 Processor -- start 2024-04-25T14:03:11.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.024+0000 7f987e053640 1 -- start start 2024-04-25T14:03:11.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f987e053640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f987815d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f987e053640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9878160c10 0x7f9878161020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f987e053640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9878161560 con 0x7f98780b1e60 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f987e053640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f98780cd630 con 0x7f9878160c10 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f987e053640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f987806e9e0 con 0x7f98780b4d50 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f9877fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9878160c10 0x7f9878161020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f9877fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9878160c10 0x7f9878161020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60084/0 (socket says 172.21.15.150:60084) 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.025+0000 7f9877fff640 1 -- 172.21.15.150:0/3389898902 learned_addr learned my addr 172.21.15.150:0/3389898902 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f98777fe640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f987815d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f9877fff640 1 -- 172.21.15.150:0/3389898902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 msgr2=0x7f987815d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f9877fff640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f987815d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f9877fff640 1 -- 172.21.15.150:0/3389898902 >> v1:172.21.15.187:6789/0 conn(0x7f98780b1e60 legacy=0x7f98780ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f9877fff640 1 -- 172.21.15.150:0/3389898902 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98780b26b0 con 0x7f9878160c10 2024-04-25T14:03:11.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.026+0000 7f9877fff640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f9878161020 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9870009e50 tx=0x7f987002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0db9a1f7f744e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f98767fc640 1 -- 172.21.15.150:0/3389898902 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870036070 con 0x7f9878160c10 2024-04-25T14:03:11.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f98767fc640 1 -- 172.21.15.150:0/3389898902 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9870031600 con 0x7f9878160c10 2024-04-25T14:03:11.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f98767fc640 1 -- 172.21.15.150:0/3389898902 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98700319d0 con 0x7f9878160c10 2024-04-25T14:03:11.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 -- 172.21.15.150:0/3389898902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 msgr2=0x7f9878161020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f9878161020 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9870009e50 tx=0x7f987002a710 comp rx=0 tx=0).stop 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 -- 172.21.15.150:0/3389898902 shutdown_connections 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f9878161020 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 --2- 172.21.15.150:0/3389898902 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f987815d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 -- 172.21.15.150:0/3389898902 >> 172.21.15.150:0/3389898902 conn(0x7f987809b4a0 msgr2=0x7f987809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 -- 172.21.15.150:0/3389898902 shutdown_connections 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.027+0000 7f987e053640 1 -- 172.21.15.150:0/3389898902 wait complete. 2024-04-25T14:03:11.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.029+0000 7f987e053640 1 Processor -- start 2024-04-25T14:03:11.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.029+0000 7f987e053640 1 -- start start 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.030+0000 7f987e053640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98780b1e60 0x7f987815fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.030+0000 7f987e053640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f98781558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f987806e670 con 0x7f9878160c10 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f987806e390 con 0x7f98780b1e60 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f98780b2270 con 0x7f98780b4d50 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60094/0 (socket says 172.21.15.150:60094) 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 -- 172.21.15.150:0/1594809032 learned_addr learned my addr 172.21.15.150:0/1594809032 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f98777fe640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98780b1e60 0x7f987815fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9876ffd640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f98781558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 -- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 msgr2=0x7f98781558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f98781558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 -- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98780b1e60 msgr2=0x7f987815fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98780b1e60 0x7f987815fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 -- 172.21.15.150:0/1594809032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9878159590 con 0x7f9878160c10 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9876ffd640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f98781558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f9877fff640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9870009f70 tx=0x7f9870004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870041020 con 0x7f9878160c10 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9870007800 con 0x7f9878160c10 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9870036040 con 0x7f9878160c10 2024-04-25T14:03:11.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9878159870 con 0x7f9878160c10 2024-04-25T14:03:11.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.031+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987815b3d0 con 0x7f9878160c10 2024-04-25T14:03:11.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.032+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9878076a40 con 0x7f9878160c10 2024-04-25T14:03:11.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.033+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9870044020 con 0x7f9878160c10 2024-04-25T14:03:11.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.033+0000 7f985ffff640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 0x7f98640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.034+0000 7f98777fe640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 0x7f98640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.034+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f98700b6310 con 0x7f9878160c10 2024-04-25T14:03:11.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.034+0000 7f98777fe640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 0x7f98640769f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f986c006fd0 tx=0x7f986c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.038+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f987004b030 con 0x7f9878160c10 2024-04-25T14:03:11.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.249+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f98780cf3b0 con 0x7f9878160c10 2024-04-25T14:03:11.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.249+0000 7f985ffff640 1 -- 172.21.15.150:0/1594809032 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f9870080730 con 0x7f9878160c10 2024-04-25T14:03:11.250 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:11.250 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:03:11.250 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 msgr2=0x7f98640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 0x7f98640769f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f986c006fd0 tx=0x7f986c008040 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 msgr2=0x7f98780d07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9870009f70 tx=0x7f9870004a40 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 shutdown_connections 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98640744e0 0x7f98640769f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9878160c10 0x7f98780d07d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98780b4d50 0x7f98781558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 --2- 172.21.15.150:0/1594809032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98780b1e60 0x7f987815fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 >> 172.21.15.150:0/1594809032 conn(0x7f987809b4a0 msgr2=0x7f98780b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.252 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 shutdown_connections 2024-04-25T14:03:11.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.251+0000 7f987e053640 1 -- 172.21.15.150:0/1594809032 wait complete. 2024-04-25T14:03:11.263 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:11.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.373+0000 7f4fe93c0640 1 Processor -- start 2024-04-25T14:03:11.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.373+0000 7f4fe93c0640 1 -- start start 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fe93c0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4fe41647a0 0x7f4fe40cfcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fe93c0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe40b5b10 0x7f4fe40b8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fe93c0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fe4164bb0 con 0x7f4fe4160070 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fe93c0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4fe40b8380 con 0x7f4fe41647a0 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fe93c0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe40d2c70 con 0x7f4fe40b5b10 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fdbfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4fe4160070 0x7f4fe40b5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46252/0 (socket says 172.21.15.150:46252) 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.374+0000 7f4fdbfff640 1 -- 172.21.15.150:0/737598507 learned_addr learned my addr 172.21.15.150:0/737598507 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe37fe640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe40b5b10 0x7f4fe40b8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe2ffd640 1 --2- 172.21.15.150:0/737598507 >> v2:172.21.15.187:3300/0 conn(0x7f4fe41647a0 0x7f4fe40cfcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2544692889 0 0) 0x7f4fe4164bb0 con 0x7f4fe4160070 2024-04-25T14:03:11.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4fd0003520 con 0x7f4fe4160070 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2824456493 0 0) 0x7f4fd0003520 con 0x7f4fe4160070 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe40b5b10 msgr2=0x7f4fe40b8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe40b5b10 0x7f4fe40b8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 >> v2:172.21.15.187:3300/0 conn(0x7f4fe41647a0 msgr2=0x7f4fe40cfcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 --2- 172.21.15.150:0/737598507 >> v2:172.21.15.187:3300/0 conn(0x7f4fe41647a0 0x7f4fe40cfcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe40d2f50 con 0x7f4fe4160070 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f4fcc001670 con 0x7f4fe4160070 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 >> v1:172.21.15.150:6789/0 conn(0x7f4fe4160070 legacy=0x7f4fe40b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fd00039b0 0x7f4fd0003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0004800 0x7f4fd0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0005150 0x7f4fd000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40b8380 con 0x7f4fd0004800 2024-04-25T14:03:11.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40d2c70 con 0x7f4fd00039b0 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fd000bf00 con 0x7f4fd0005150 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0004800 0x7f4fd0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe2ffd640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fd00039b0 0x7f4fd0003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fe37fe640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0005150 0x7f4fd000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 -- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0005150 msgr2=0x7f4fd000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0005150 0x7f4fd000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 -- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fd00039b0 msgr2=0x7f4fd0003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fd00039b0 0x7f4fd0003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.375+0000 7f4fdbfff640 1 -- 172.21.15.150:0/737598507 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fd0003520 con 0x7f4fd0004800 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.376+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0004800 0x7f4fd0004c10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b6bd0 tx=0x7f4fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ba323797863cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.376+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc007d50 con 0x7f4fd0004800 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.376+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fcc005ad0 con 0x7f4fd0004800 2024-04-25T14:03:11.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.376+0000 7f4fe27fc640 1 -- 172.21.15.150:0/737598507 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc007070 con 0x7f4fd0004800 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 -- 172.21.15.150:0/737598507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0004800 msgr2=0x7f4fd0004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/737598507 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fd0004800 0x7f4fd0004c10 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40b6bd0 tx=0x7f4fd0005830 comp rx=0 tx=0).stop 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe37fe640 1 -- 172.21.15.150:0/737598507 reap_dead start 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 -- 172.21.15.150:0/737598507 shutdown_connections 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 -- 172.21.15.150:0/737598507 >> 172.21.15.150:0/737598507 conn(0x7f4fe40d0400 msgr2=0x7f4fe40d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 -- 172.21.15.150:0/737598507 shutdown_connections 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.377+0000 7f4fe93c0640 1 -- 172.21.15.150:0/737598507 wait complete. 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.379+0000 7f4fe93c0640 1 Processor -- start 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.379+0000 7f4fe93c0640 1 -- start start 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0004800 0x7f4fe41575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 0x7f4fe4159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40cdfd0 con 0x7f4fe4161f50 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe40b7fa0 con 0x7f4fe4154a20 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe93c0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe40ce150 con 0x7f4fd0004800 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.380+0000 7f4fe37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60118/0 (socket says 172.21.15.150:60118) 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 -- 172.21.15.150:0/1619812258 learned_addr learned my addr 172.21.15.150:0/1619812258 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe2ffd640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0004800 0x7f4fe41575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 0x7f4fe4159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 -- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0004800 msgr2=0x7f4fe41575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0004800 0x7f4fe41575b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 -- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 msgr2=0x7f4fe4159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 0x7f4fe4159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 -- 172.21.15.150:0/1619812258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe4155190 con 0x7f4fe4161f50 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fdbfff640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 0x7f4fe4159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:11.384 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe37fe640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40692d0 tx=0x7f4fdc00f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00fce0 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fdc01d070 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe4155470 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc015670 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.381+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe4155a30 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.382+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fe4070e80 con 0x7f4fe4161f50 2024-04-25T14:03:11.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.383+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4fdc015810 con 0x7f4fe4161f50 2024-04-25T14:03:11.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.383+0000 7f4fdb7fe640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 0x7f4fc40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.383+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f4fdc095e20 con 0x7f4fe4161f50 2024-04-25T14:03:11.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.383+0000 7f4fe2ffd640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 0x7f4fc40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.386 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.384+0000 7f4fe2ffd640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 0x7f4fc40769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009bc0 tx=0x7f4fd4009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.390+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fe4070e80 con 0x7f4fe4161f50 2024-04-25T14:03:11.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.600+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4fe4076a40 con 0x7f4fe4161f50 2024-04-25T14:03:11.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.601+0000 7f4fdb7fe640 1 -- 172.21.15.150:0/1619812258 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5186 (secure 0 0 0) 0x7f4fdc010040 con 0x7f4fe4161f50 2024-04-25T14:03:11.601 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:11.601 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/443710057","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":443710057},{"type":"v1","addr":"172.21.15.150:6837","nonce":443710057}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4229,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4100664268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4100664268},{"type":"v1","addr":"172.21.15.187:6834","nonce":4100664268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/4291646086","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":4291646086},{"type":"v1","addr":"172.21.15.150:6835","nonce":4291646086}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:02:08.490370+0000","modified":"2024-04-25T14:02:30.598800+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/3860691734","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3860691734},{"type":"v1","addr":"172.21.15.187:6835","nonce":3860691734}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-25T14:03:11.602 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 8 2024-04-25T14:03:11.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 msgr2=0x7f4fc40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 0x7f4fc40769c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009bc0 tx=0x7f4fd4009190 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 msgr2=0x7f4fe4156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40692d0 tx=0x7f4fdc00f1b0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 shutdown_connections 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fc40744b0 0x7f4fc40769c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe4161f50 0x7f4fe4156230 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe4154a20 0x7f4fe4159020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 --2- 172.21.15.150:0/1619812258 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fd0004800 0x7f4fe41575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 >> 172.21.15.150:0/1619812258 conn(0x7f4fe40d0400 msgr2=0x7f4fe40cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 shutdown_connections 2024-04-25T14:03:11.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.603+0000 7f4fe93c0640 1 -- 172.21.15.150:0/1619812258 wait complete. 2024-04-25T14:03:11.615 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:11.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.721+0000 7feab3a34640 1 Processor -- start 2024-04-25T14:03:11.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.721+0000 7feab3a34640 1 -- start start 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab3a34640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaac160210 0x7feaac0d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab3a34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac0b5ab0 0x7feaac0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab3a34640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaac0d20a0 con 0x7feaac0b2ba0 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab3a34640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7feaac0d2380 con 0x7feaac160210 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab3a34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feaac0cfc30 con 0x7feaac0b5ab0 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab17d0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7feaac160210 0x7feaac0d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab0fcf640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7feaac0b2ba0 0x7feaac0b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46254/0 (socket says 172.21.15.150:46254) 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab0fcf640 1 -- 172.21.15.150:0/1844912009 learned_addr learned my addr 172.21.15.150:0/1844912009 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab1fd1640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac0b5ab0 0x7feaac0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab17d0640 1 -- 172.21.15.150:0/1844912009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac0b5ab0 msgr2=0x7feaac0cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:11.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab17d0640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac0b5ab0 0x7feaac0cdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feaa3fff640 1 -- 172.21.15.150:0/1844912009 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 530310766 0 0) 0x7feaac0d20a0 con 0x7feaac0b2ba0 2024-04-25T14:03:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab17d0640 1 -- 172.21.15.150:0/1844912009 >> v1:172.21.15.150:6789/0 conn(0x7feaac0b2ba0 legacy=0x7feaac0b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.723+0000 7feab17d0640 1 -- 172.21.15.150:0/1844912009 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac0d0150 con 0x7feaac160210 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.724+0000 7feab17d0640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac160210 0x7feaac0d2bd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feaa80097a0 tx=0x7feaa802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ea06492edfefc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.724+0000 7feaa3fff640 1 -- 172.21.15.150:0/1844912009 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8037070 con 0x7feaac160210 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.724+0000 7feaa3fff640 1 -- 172.21.15.150:0/1844912009 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa8032440 con 0x7feaac160210 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.724+0000 7feaa3fff640 1 -- 172.21.15.150:0/1844912009 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa8032810 con 0x7feaac160210 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 -- 172.21.15.150:0/1844912009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac160210 msgr2=0x7feaac0d2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac160210 0x7feaac0d2bd0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7feaa80097a0 tx=0x7feaa802a180 comp rx=0 tx=0).stop 2024-04-25T14:03:11.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 -- 172.21.15.150:0/1844912009 shutdown_connections 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac0b5ab0 0x7feaac0cdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 --2- 172.21.15.150:0/1844912009 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac160210 0x7feaac0d2bd0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 -- 172.21.15.150:0/1844912009 >> 172.21.15.150:0/1844912009 conn(0x7feaac0d03a0 msgr2=0x7feaac0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 -- 172.21.15.150:0/1844912009 shutdown_connections 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.725+0000 7feab3a34640 1 -- 172.21.15.150:0/1844912009 wait complete. 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.727+0000 7feab3a34640 1 Processor -- start 2024-04-25T14:03:11.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.727+0000 7feab3a34640 1 -- start start 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaac0b2ba0 0x7feaac15a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac160210 0x7feaac0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feaac0ce630 con 0x7feaac0b48f0 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feaac0ce4b0 con 0x7feaac0b2ba0 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab3a34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feaac0ce7b0 con 0x7feaac160210 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab1fd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab1fd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60142/0 (socket says 172.21.15.150:60142) 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab1fd1640 1 -- 172.21.15.150:0/1439399229 learned_addr learned my addr 172.21.15.150:0/1439399229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab0fcf640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac160210 0x7feaac0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.728+0000 7feab17d0640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaac0b2ba0 0x7feaac15a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 -- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac160210 msgr2=0x7feaac0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac160210 0x7feaac0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 -- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaac0b2ba0 msgr2=0x7feaac15a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaac0b2ba0 0x7feaac15a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 -- 172.21.15.150:0/1439399229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaac0b5030 con 0x7feaac0b48f0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab1fd1640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7feaa400efc0 tx=0x7feaa400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4018020 con 0x7feaac0b48f0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa4014480 con 0x7feaac0b48f0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4009e60 con 0x7feaac0b48f0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feaac06f200 con 0x7feaac0b48f0 2024-04-25T14:03:11.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.729+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feaac0ca990 con 0x7feaac0b48f0 2024-04-25T14:03:11.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.730+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feaac0b4da0 con 0x7feaac0b48f0 2024-04-25T14:03:11.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.731+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feaa4005d10 con 0x7feaac0b48f0 2024-04-25T14:03:11.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.731+0000 7feaa1ffb640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 0x7fea900769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:11.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.731+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7feaa4096280 con 0x7feaac0b48f0 2024-04-25T14:03:11.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.731+0000 7feab17d0640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 0x7fea900769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:11.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.732+0000 7feab17d0640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 0x7fea900769f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feaa8034000 tx=0x7feaa802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:11.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.736+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feaa4022030 con 0x7feaac0b48f0 2024-04-25T14:03:11.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.913+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feaac076a40 con 0x7feaac0b48f0 2024-04-25T14:03:11.914 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.914+0000 7feaa1ffb640 1 -- 172.21.15.150:0/1439399229 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18902 (secure 0 0 0) 0x7feaa4010040 con 0x7feaac0b48f0 2024-04-25T14:03:11.914 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:11.915 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:02:50.289420+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:02:06.806760+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T14:02:07.444080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-25T14:02:11.170027+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.b","mappings":[{"from":3,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.3","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1d","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:11.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 msgr2=0x7fea900769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.916 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 0x7fea900769f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feaa8034000 tx=0x7feaa802e000 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 msgr2=0x7feaac0c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7feaa400efc0 tx=0x7feaa400c490 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 shutdown_connections 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fea900744e0 0x7fea900769f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feaac0b48f0 0x7feaac0c9800 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feaac160210 0x7feaac0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 --2- 172.21.15.150:0/1439399229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feaac0b2ba0 0x7feaac15a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 >> 172.21.15.150:0/1439399229 conn(0x7feaac0d03a0 msgr2=0x7feaac0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 shutdown_connections 2024-04-25T14:03:11.917 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:11.916+0000 7feab3a34640 1 -- 172.21.15.150:0/1439399229 wait complete. 2024-04-25T14:03:11.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T14:03:12.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.035+0000 7f6d3ee53640 1 Processor -- start 2024-04-25T14:03:12.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.035+0000 7f6d3ee53640 1 -- start start 2024-04-25T14:03:12.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.036+0000 7f6d3ee53640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d38072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.036+0000 7f6d3ee53640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f6d38162300 0x7f6d380733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.036+0000 7f6d3ee53640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d3815e7f0 con 0x7f6d380b2f20 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.036+0000 7f6d3ee53640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f6d3815ee20 con 0x7f6d38162300 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.036+0000 7f6d3ee53640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d38162920 con 0x7f6d3815f2b0 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d37fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f6d380b2f20 0x7f6d38161bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46268/0 (socket says 172.21.15.150:46268) 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d37fff640 1 -- 172.21.15.150:0/2908417711 learned_addr learned my addr 172.21.15.150:0/2908417711 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3cbef640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d38072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3d3f0640 1 --2- 172.21.15.150:0/2908417711 >> v2:172.21.15.187:3300/0 conn(0x7f6d38162300 0x7f6d380733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d377fe640 1 -- 172.21.15.150:0/2908417711 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3280889276 0 0) 0x7f6d3815e7f0 con 0x7f6d380b2f20 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d377fe640 1 -- 172.21.15.150:0/2908417711 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6d20003520 con 0x7f6d380b2f20 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3d3f0640 1 -- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 msgr2=0x7f6d38072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3d3f0640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d38072ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3d3f0640 1 -- 172.21.15.150:0/2908417711 >> v1:172.21.15.150:6789/0 conn(0x7f6d380b2f20 legacy=0x7f6d38161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.037+0000 7f6d3d3f0640 1 -- 172.21.15.150:0/2908417711 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d3815eab0 con 0x7f6d38162300 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.038+0000 7f6d3d3f0640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380733e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6d3815efa0 tx=0x7f6d3002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b742ae0818572f7b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.038+0000 7f6d377fe640 1 -- 172.21.15.150:0/2908417711 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30031600 con 0x7f6d38162300 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.038+0000 7f6d377fe640 1 -- 172.21.15.150:0/2908417711 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d300317a0 con 0x7f6d38162300 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.038+0000 7f6d377fe640 1 -- 172.21.15.150:0/2908417711 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30031b50 con 0x7f6d38162300 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 -- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 msgr2=0x7f6d380733e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380733e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6d3815efa0 tx=0x7f6d3002a5d0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 -- 172.21.15.150:0/2908417711 shutdown_connections 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380733e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/2908417711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d38072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 -- 172.21.15.150:0/2908417711 >> 172.21.15.150:0/2908417711 conn(0x7f6d380d39b0 msgr2=0x7f6d380d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:12.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 -- 172.21.15.150:0/2908417711 shutdown_connections 2024-04-25T14:03:12.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.039+0000 7f6d3ee53640 1 -- 172.21.15.150:0/2908417711 wait complete. 2024-04-25T14:03:12.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.041+0000 7f6d3ee53640 1 Processor -- start 2024-04-25T14:03:12.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.041+0000 7f6d3ee53640 1 -- start start 2024-04-25T14:03:12.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d380c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d380d26b0 con 0x7f6d380b2f20 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d380d2530 con 0x7f6d38162300 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3ee53640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d380d2830 con 0x7f6d3815f2b0 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3cbef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3cbef640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60146/0 (socket says 172.21.15.150:60146) 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.042+0000 7f6d3cbef640 1 -- 172.21.15.150:0/1187673265 learned_addr learned my addr 172.21.15.150:0/1187673265 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3d3f0640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d37fff640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d380c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 -- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 msgr2=0x7f6d380c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d380c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.044 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 -- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 msgr2=0x7f6d380c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 -- 172.21.15.150:0/1187673265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d380ca8c0 con 0x7f6d380b2f20 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3d3f0640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3cbef640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e870 tx=0x7f6d2800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d28017070 con 0x7f6d380b2f20 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2800c940 con 0x7f6d380b2f20 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d381e7f50 con 0x7f6d380b2f20 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800cd10 con 0x7f6d380b2f20 2024-04-25T14:03:12.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.043+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d381e8510 con 0x7f6d380b2f20 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.044+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d38076a40 con 0x7f6d380b2f20 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.045+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6d2801a070 con 0x7f6d380b2f20 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.045+0000 7f6d357fa640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 0x7f6d240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.045+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f6d28095860 con 0x7f6d380b2f20 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.045+0000 7f6d37fff640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 0x7f6d240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.046+0000 7f6d37fff640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 0x7f6d240769f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6d380749b0 tx=0x7f6d2c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.050+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d2805fbf0 con 0x7f6d380b2f20 2024-04-25T14:03:12.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.266+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6d380d1e00 con 0x7f6d380b2f20 2024-04-25T14:03:12.267 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:03:12.267+0000 7f46d47d6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:03:12.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.269+0000 7f6d357fa640 1 -- 172.21.15.150:0/1187673265 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7f6d2805f4f0 con 0x7f6d380b2f20 2024-04-25T14:03:12.269 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.271+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 msgr2=0x7f6d240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.271+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 0x7f6d240769f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6d380749b0 tx=0x7f6d2c009040 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.271+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 msgr2=0x7f6d3814bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.271+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e870 tx=0x7f6d2800ed30 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 shutdown_connections 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d240744e0 0x7f6d240769f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d38162300 0x7f6d380c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d3815f2b0 0x7f6d380c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 --2- 172.21.15.150:0/1187673265 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d380b2f20 0x7f6d3814bc10 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 >> 172.21.15.150:0/1187673265 conn(0x7f6d380d39b0 msgr2=0x7f6d3815cbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:12.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 shutdown_connections 2024-04-25T14:03:12.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.272+0000 7f6d3ee53640 1 -- 172.21.15.150:0/1187673265 wait complete. 2024-04-25T14:03:12.276 INFO:tasks.ceph.mds.b.smithi187.stderr: -5450> 2024-04-25T14:02:13.299+0000 7f5757ae9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:03:12.284 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T14:03:12.333 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:03:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.397+0000 7fc8fcd72640 1 Processor -- start 2024-04-25T14:03:12.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.397+0000 7fc8fcd72640 1 -- start start 2024-04-25T14:03:12.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8fcd72640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8fcd72640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 0x7fc8f806f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8fcd72640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8f80cf300 con 0x7fc8f8070820 2024-04-25T14:03:12.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8fcd72640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc8f80cff10 con 0x7fc8f81610f0 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8fcd72640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8f80cf5e0 con 0x7fc8f80d03a0 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.398+0000 7fc8f5d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc8f8070820 0x7fc8f81609e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46280/0 (socket says 172.21.15.150:46280) 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f5d9b640 1 -- 172.21.15.150:0/58843716 learned_addr learned my addr 172.21.15.150:0/58843716 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f659c640 1 --2- 172.21.15.150:0/58843716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f6d9d640 1 --2- 172.21.15.150:0/58843716 >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 0x7fc8f806f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282143003 0 0) 0x7fc8f80cf300 con 0x7fc8f8070820 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc8e0003520 con 0x7fc8f8070820 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f659c640 1 -- 172.21.15.150:0/58843716 >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 msgr2=0x7fc8f806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1428999418 0 0) 0x7fc8e0003520 con 0x7fc8f8070820 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f659c640 1 --2- 172.21.15.150:0/58843716 >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 0x7fc8f806f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f659c640 1 -- 172.21.15.150:0/58843716 >> v1:172.21.15.150:6789/0 conn(0x7fc8f8070820 legacy=0x7fc8f81609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f659c640 1 -- 172.21.15.150:0/58843716 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f80cf8c0 con 0x7fc8f80d03a0 2024-04-25T14:03:12.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.399+0000 7fc8f6d9d640 1 --2- 172.21.15.150:0/58843716 >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 0x7fc8f806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.400+0000 7fc8f659c640 1 --2- 172.21.15.150:0/58843716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80cbd80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80097a0 tx=0x7fc8e802a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74a77d086d673ddc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.400+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8e8037070 con 0x7fc8f80d03a0 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.400+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8e802aba0 con 0x7fc8f80d03a0 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.400+0000 7fc8f559a640 1 -- 172.21.15.150:0/58843716 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec001670 con 0x7fc8f80d03a0 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 -- 172.21.15.150:0/58843716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 msgr2=0x7fc8f80cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/58843716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80cbd80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc8e80097a0 tx=0x7fc8e802a110 comp rx=0 tx=0).stop 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 -- 172.21.15.150:0/58843716 shutdown_connections 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/58843716 >> v2:172.21.15.187:3300/0 conn(0x7fc8f81610f0 0x7fc8f806f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/58843716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80cbd80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 -- 172.21.15.150:0/58843716 >> 172.21.15.150:0/58843716 conn(0x7fc8f80ce390 msgr2=0x7fc8f80ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:12.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 -- 172.21.15.150:0/58843716 shutdown_connections 2024-04-25T14:03:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.401+0000 7fc8fcd72640 1 -- 172.21.15.150:0/58843716 wait complete. 2024-04-25T14:03:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.403+0000 7fc8fcd72640 1 Processor -- start 2024-04-25T14:03:12.403 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.403+0000 7fc8fcd72640 1 -- start start 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f8070820 0x7fc8f80b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f80cdca0 con 0x7fc8f80b1d50 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc8f80cc2c0 con 0x7fc8f8070820 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8fcd72640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc8f80cde20 con 0x7fc8f80d03a0 2024-04-25T14:03:12.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60156/0 (socket says 172.21.15.150:60156) 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 -- 172.21.15.150:0/926166340 learned_addr learned my addr 172.21.15.150:0/926166340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f659c640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f8070820 0x7fc8f80b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f5d9b640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 -- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 msgr2=0x7fc8f80b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 -- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f8070820 msgr2=0x7fc8f80b5c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f8070820 0x7fc8f80b5c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 -- 172.21.15.150:0/926166340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8f814cd20 con 0x7fc8f80b1d50 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8f6d9d640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000efb0 tx=0x7fc8f000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0018020 con 0x7fc8f80b1d50 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.405+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8f0014480 con 0x7fc8f80b1d50 2024-04-25T14:03:12.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.406+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f0014850 con 0x7fc8f80b1d50 2024-04-25T14:03:12.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.406+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8f80b2940 con 0x7fc8f80b1d50 2024-04-25T14:03:12.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.406+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8f80b2f00 con 0x7fc8f80b1d50 2024-04-25T14:03:12.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.407+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8f8076a40 con 0x7fc8f80b1d50 2024-04-25T14:03:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.407+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc8f0005d10 con 0x7fc8f80b1d50 2024-04-25T14:03:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.408+0000 7fc8e6ffd640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 0x7fc8d80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.408+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fc8f0096230 con 0x7fc8f80b1d50 2024-04-25T14:03:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.408+0000 7fc8f659c640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 0x7fc8d80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:12.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.408+0000 7fc8f659c640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 0x7fc8d80769c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8034000 tx=0x7fc8e802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:12.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.412+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc8f0022030 con 0x7fc8f80b1d50 2024-04-25T14:03:12.635 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:12.635+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fc8f806c880 con 0x7fc8f80b1d50 2024-04-25T14:03:13.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.270+0000 7fc8e6ffd640 1 -- 172.21.15.150:0/926166340 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7fc8f0010040 con 0x7fc8f80b1d50 2024-04-25T14:03:13.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 msgr2=0x7fc8d80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 0x7fc8d80769c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8034000 tx=0x7fc8e802e000 comp rx=0 tx=0).stop 2024-04-25T14:03:13.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 msgr2=0x7fc8f8160390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc8f000efb0 tx=0x7fc8f000c490 comp rx=0 tx=0).stop 2024-04-25T14:03:13.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 shutdown_connections 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc8d80744b0 0x7fc8d80769c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc8f80b1d50 0x7fc8f8160390 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc8f80d03a0 0x7fc8f80b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 --2- 172.21.15.150:0/926166340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc8f8070820 0x7fc8f80b5c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 >> 172.21.15.150:0/926166340 conn(0x7fc8f80ce390 msgr2=0x7fc8f806d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 shutdown_connections 2024-04-25T14:03:13.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.273+0000 7fc8fcd72640 1 -- 172.21.15.150:0/926166340 wait complete. 2024-04-25T14:03:13.285 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T14:03:13.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.399+0000 7fa396eb8640 1 Processor -- start 2024-04-25T14:03:13.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.399+0000 7fa396eb8640 1 -- start start 2024-04-25T14:03:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.400+0000 7fa396eb8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa390070d90 0x7fa390161510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.400+0000 7fa396eb8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa396eb8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa39009f920 con 0x7fa3900739c0 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa396eb8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa390160a70 con 0x7fa390070d90 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa396eb8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa3900b0be0 con 0x7fa39015cd20 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa394c54640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa390070d90 0x7fa390161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa38ffff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:52332/0 (socket says 172.21.15.150:52332) 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa38ffff640 1 -- 172.21.15.150:0/836257391 learned_addr learned my addr 172.21.15.150:0/836257391 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa395455640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fa3900739c0 0x7fa390073dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:40744/0 (socket says 172.21.15.150:40744) 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa394c54640 1 -- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 msgr2=0x7fa3900b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa394c54640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa394c54640 1 -- 172.21.15.150:0/836257391 >> v1:172.21.15.187:6789/0 conn(0x7fa3900739c0 legacy=0x7fa390073dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa394c54640 1 -- 172.21.15.150:0/836257391 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3900b0ea0 con 0x7fa390070d90 2024-04-25T14:03:13.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.401+0000 7fa38ffff640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.402+0000 7fa394c54640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390161510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3840097a0 tx=0x7fa38402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d484bea2dbefee99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.402+0000 7fa38f7fe640 1 -- 172.21.15.150:0/836257391 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa384037070 con 0x7fa390070d90 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.402+0000 7fa38f7fe640 1 -- 172.21.15.150:0/836257391 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa384032440 con 0x7fa390070d90 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.402+0000 7fa38f7fe640 1 -- 172.21.15.150:0/836257391 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa384032810 con 0x7fa390070d90 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 -- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 msgr2=0x7fa390161510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390161510 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa3840097a0 tx=0x7fa38402a180 comp rx=0 tx=0).stop 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 -- 172.21.15.150:0/836257391 shutdown_connections 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa39015cd20 0x7fa3900b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 --2- 172.21.15.150:0/836257391 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390161510 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 -- 172.21.15.150:0/836257391 >> 172.21.15.150:0/836257391 conn(0x7fa3900d1f80 msgr2=0x7fa3900732b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 -- 172.21.15.150:0/836257391 shutdown_connections 2024-04-25T14:03:13.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.403+0000 7fa396eb8640 1 -- 172.21.15.150:0/836257391 wait complete. 2024-04-25T14:03:13.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.405+0000 7fa396eb8640 1 Processor -- start 2024-04-25T14:03:13.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.405+0000 7fa396eb8640 1 -- start start 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390156750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3900739c0 0x7fa3900b4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa3900b1180 con 0x7fa39015cd20 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa390160ff0 con 0x7fa390070d90 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa396eb8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa39015d130 con 0x7fa3900739c0 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa395455640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa395455640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60158/0 (socket says 172.21.15.150:60158) 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa395455640 1 -- 172.21.15.150:0/3931829854 learned_addr learned my addr 172.21.15.150:0/3931829854 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa38ffff640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3900739c0 0x7fa3900b4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.406+0000 7fa394c54640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390156750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 -- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3900739c0 msgr2=0x7fa3900b4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3900739c0 0x7fa3900b4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 -- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 msgr2=0x7fa390156750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390156750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 -- 172.21.15.150:0/3931829854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3901437f0 con 0x7fa39015cd20 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa395455640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c6f0 tx=0x7fa38800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa388017070 con 0x7fa39015cd20 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa388005da0 con 0x7fa39015cd20 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3900b55e0 con 0x7fa39015cd20 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa38800a840 con 0x7fa39015cd20 2024-04-25T14:03:13.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.407+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa39015ba50 con 0x7fa39015cd20 2024-04-25T14:03:13.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.408+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3900b5350 con 0x7fa39015cd20 2024-04-25T14:03:13.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.409+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa38801a070 con 0x7fa39015cd20 2024-04-25T14:03:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.409+0000 7fa38d7fa640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 0x7fa3740769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.410+0000 7fa394c54640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 0x7fa3740769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.410+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6821+0+0 (secure 0 0 0) 0x7fa388094ac0 con 0x7fa39015cd20 2024-04-25T14:03:13.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.410+0000 7fa394c54640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 0x7fa3740769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa384034000 tx=0x7fa38402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:13.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.414+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa3901437f0 con 0x7fa39015cd20 2024-04-25T14:03:13.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:13.597+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa390076a40 con 0x7fa39015cd20 2024-04-25T14:03:14.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.393+0000 7fa38d7fa640 1 -- 172.21.15.150:0/3931829854 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v28) ==== 181+0+0 (secure 0 0 0) 0x7fa388065e20 con 0x7fa39015cd20 2024-04-25T14:03:14.393 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-04-25T14:03:14.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 msgr2=0x7fa3740769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 0x7fa3740769c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa384034000 tx=0x7fa38402e000 comp rx=0 tx=0).stop 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 msgr2=0x7fa3900b4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa38800c6f0 tx=0x7fa38800cbb0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 shutdown_connections 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa3740744b0 0x7fa3740769c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa39015cd20 0x7fa3900b4ca0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa3900739c0 0x7fa3900b4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 --2- 172.21.15.150:0/3931829854 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa390070d90 0x7fa390156750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 >> 172.21.15.150:0/3931829854 conn(0x7fa3900d1f80 msgr2=0x7fa39014ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 shutdown_connections 2024-04-25T14:03:14.397 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.396+0000 7fa396eb8640 1 -- 172.21.15.150:0/3931829854 wait complete. 2024-04-25T14:03:14.416 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T14:03:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.533+0000 7f6369acd640 1 Processor -- start 2024-04-25T14:03:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.533+0000 7f6369acd640 1 -- start start 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f6369acd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63640734f0 0x7f63640d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f6369acd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63640b70b0 con 0x7f636415e110 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f6369acd640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63640ce930 con 0x7f63640b6ca0 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f6369acd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f63640cec00 con 0x7f63640734f0 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f6363fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f63640734f0 0x7f63640d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f63637fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f636415e110 0x7f63640ce220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46284/0 (socket says 172.21.15.150:46284) 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.534+0000 7f63637fe640 1 -- 172.21.15.150:0/1617247011 learned_addr learned my addr 172.21.15.150:0/1617247011 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f6363fff640 1 -- 172.21.15.150:0/1617247011 >> v1:172.21.15.187:6789/0 conn(0x7f63640b6ca0 legacy=0x7f6364072de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f6363fff640 1 -- 172.21.15.150:0/1617247011 >> v1:172.21.15.150:6789/0 conn(0x7f636415e110 legacy=0x7f63640ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f6363fff640 1 -- 172.21.15.150:0/1617247011 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63640ceee0 con 0x7f63640734f0 2024-04-25T14:03:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f63627fc640 1 -- 172.21.15.150:0/1617247011 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 561218898 0 0) 0x7f63640b70b0 con 0x7f636415e110 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f6363fff640 1 --2- 172.21.15.150:0/1617247011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63640734f0 0x7f63640d2840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778ebcf619f643fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f63627fc640 1 -- 172.21.15.150:0/1617247011 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c031600 con 0x7f63640734f0 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.535+0000 7f63627fc640 1 -- 172.21.15.150:0/1617247011 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635c0317a0 con 0x7f63640734f0 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f63627fc640 1 -- 172.21.15.150:0/1617247011 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635c031b70 con 0x7f63640734f0 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 -- 172.21.15.150:0/1617247011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63640734f0 msgr2=0x7f63640d2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 --2- 172.21.15.150:0/1617247011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63640734f0 0x7f63640d2840 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f635c009d90 tx=0x7f635c02a610 comp rx=0 tx=0).stop 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 -- 172.21.15.150:0/1617247011 shutdown_connections 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 --2- 172.21.15.150:0/1617247011 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f63640734f0 0x7f63640d2840 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 -- 172.21.15.150:0/1617247011 >> 172.21.15.150:0/1617247011 conn(0x7f63640d39c0 msgr2=0x7f63640d3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 -- 172.21.15.150:0/1617247011 shutdown_connections 2024-04-25T14:03:14.537 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.536+0000 7f6369acd640 1 -- 172.21.15.150:0/1617247011 wait complete. 2024-04-25T14:03:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.538+0000 7f6369acd640 1 Processor -- start 2024-04-25T14:03:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.538+0000 7f6369acd640 1 -- start start 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63640b6ca0 0x7f636414e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f636415c160 0x7f636415c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f63640d2320 con 0x7f636415e110 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f63640d2040 con 0x7f636415c160 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6369acd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6364161bd0 con 0x7f63640b6ca0 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.539+0000 7f6362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60172/0 (socket says 172.21.15.150:60172) 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 -- 172.21.15.150:0/509243408 learned_addr learned my addr 172.21.15.150:0/509243408 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6363fff640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f636415c160 0x7f636415c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f63637fe640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63640b6ca0 0x7f636414e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 -- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63640b6ca0 msgr2=0x7f636414e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63640b6ca0 0x7f636414e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 -- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f636415c160 msgr2=0x7f636415c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f636415c160 0x7f636415c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 -- 172.21.15.150:0/509243408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f636415cab0 con 0x7f636415e110 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6362ffd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f635800b410 tx=0x7f635800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358015020 con 0x7f636415e110 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6358011830 con 0x7f636415e110 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63640c7c00 con 0x7f636415e110 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6358011c20 con 0x7f636415e110 2024-04-25T14:03:14.542 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.540+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63641f5ab0 con 0x7f636415e110 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.541+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63640aabb0 con 0x7f636415e110 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.542+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6358018020 con 0x7f636415e110 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.542+0000 7f634bfff640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 0x7f63500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.542+0000 7f63637fe640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 0x7f63500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.542+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6383+0+0 (secure 0 0 0) 0x7f6358095300 con 0x7f636415e110 2024-04-25T14:03:14.543 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.543+0000 7f63637fe640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 0x7f63500769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6354007fd0 tx=0x7f6354007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:14.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.547+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f635801f020 con 0x7f636415e110 2024-04-25T14:03:14.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:14.724+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6364076a40 con 0x7f636415e110 2024-04-25T14:03:15.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.410+0000 7f634bfff640 1 -- 172.21.15.150:0/509243408 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v29) ==== 169+0+0 (secure 0 0 0) 0x7f636415cab0 con 0x7f636415e110 2024-04-25T14:03:15.410 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.411+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 msgr2=0x7f63500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.411+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 0x7f63500769f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6354007fd0 tx=0x7f6354007de0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.411+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 msgr2=0x7f63640aa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.411+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f635800b410 tx=0x7f635800b8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 shutdown_connections 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f63500744e0 0x7f63500769f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f636415c160 0x7f636415c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f636415e110 0x7f63640aa3d0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 --2- 172.21.15.150:0/509243408 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f63640b6ca0 0x7f636414e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 >> 172.21.15.150:0/509243408 conn(0x7f63640d39c0 msgr2=0x7f63640b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 shutdown_connections 2024-04-25T14:03:15.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.412+0000 7f6369acd640 1 -- 172.21.15.150:0/509243408 wait complete. 2024-04-25T14:03:15.428 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-25T14:03:15.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.550+0000 7f540e372640 1 Processor -- start 2024-04-25T14:03:15.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.550+0000 7f540e372640 1 -- start start 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540e372640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5408163760 0x7f540806cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540e372640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54080d13f0 0x7f54080b9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540e372640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5408163b70 con 0x7f540806d4c0 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540e372640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f540806d8d0 con 0x7f54080d13f0 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540e372640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f54080b9a40 con 0x7f5408163760 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54080d13f0 0x7f54080b9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f54080d13f0 0x7f54080b9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60188/0 (socket says 172.21.15.150:60188) 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 -- 172.21.15.150:0/2182179045 learned_addr learned my addr 172.21.15.150:0/2182179045 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f5407fff640 1 --2- 172.21.15.150:0/2182179045 >> v2:172.21.15.187:3300/0 conn(0x7f5408163760 0x7f540806cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 -- 172.21.15.150:0/2182179045 >> v2:172.21.15.187:3300/0 conn(0x7f5408163760 msgr2=0x7f540806cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 --2- 172.21.15.150:0/2182179045 >> v2:172.21.15.187:3300/0 conn(0x7f5408163760 0x7f540806cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.551+0000 7f540c90f640 1 -- 172.21.15.150:0/2182179045 >> v1:172.21.15.187:6789/0 conn(0x7f540806d4c0 legacy=0x7f54080d0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:15.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.552+0000 7f540c90f640 1 -- 172.21.15.150:0/2182179045 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f540806daf0 con 0x7f54080d13f0 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.552+0000 7f540c90f640 1 --2- 172.21.15.150:0/2182179045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54080d13f0 0x7f54080b9ed0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f540808e5a0 tx=0x7f540002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4b3948ec0f136f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.552+0000 7f5406ffd640 1 -- 172.21.15.150:0/2182179045 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400036070 con 0x7f54080d13f0 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.552+0000 7f5406ffd640 1 -- 172.21.15.150:0/2182179045 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5400031600 con 0x7f54080d13f0 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.552+0000 7f5406ffd640 1 -- 172.21.15.150:0/2182179045 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54000319d0 con 0x7f54080d13f0 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 -- 172.21.15.150:0/2182179045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54080d13f0 msgr2=0x7f54080b9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 --2- 172.21.15.150:0/2182179045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54080d13f0 0x7f54080b9ed0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f540808e5a0 tx=0x7f540002a660 comp rx=0 tx=0).stop 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 -- 172.21.15.150:0/2182179045 shutdown_connections 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 --2- 172.21.15.150:0/2182179045 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f54080d13f0 0x7f54080b9ed0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 --2- 172.21.15.150:0/2182179045 >> v2:172.21.15.187:3300/0 conn(0x7f5408163760 0x7f540806cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 -- 172.21.15.150:0/2182179045 >> 172.21.15.150:0/2182179045 conn(0x7f54080d2720 msgr2=0x7f54080d2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 -- 172.21.15.150:0/2182179045 shutdown_connections 2024-04-25T14:03:15.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.553+0000 7f540e372640 1 -- 172.21.15.150:0/2182179045 wait complete. 2024-04-25T14:03:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.555+0000 7f540e372640 1 Processor -- start 2024-04-25T14:03:15.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.555+0000 7f540e372640 1 -- start start 2024-04-25T14:03:15.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f540806d4c0 0x7f540814d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 0x7f5408149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f54080d4230 con 0x7f5408163760 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f540806de90 con 0x7f540806d4c0 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.556+0000 7f540e372640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f54080b92f0 con 0x7f54080d13f0 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60192/0 (socket says 172.21.15.150:60192) 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 -- 172.21.15.150:0/2400154322 learned_addr learned my addr 172.21.15.150:0/2400154322 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f5407fff640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f540806d4c0 0x7f540814d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f54077fe640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 0x7f5408149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 -- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 msgr2=0x7f5408149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:15.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 0x7f5408149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 -- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f540806d4c0 msgr2=0x7f540814d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f540806d4c0 0x7f540814d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 -- 172.21.15.150:0/2400154322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5408159740 con 0x7f5408163760 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f54077fe640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 0x7f5408149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540c90f640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f540002ac50 tx=0x7f5400004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400036050 con 0x7f5408163760 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5400042070 con 0x7f5408163760 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f540815a460 con 0x7f5408163760 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5400009510 con 0x7f5408163760 2024-04-25T14:03:15.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.557+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f540815aa20 con 0x7f5408163760 2024-04-25T14:03:15.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.558+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5408076a40 con 0x7f5408163760 2024-04-25T14:03:15.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.559+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5400007c30 con 0x7f5408163760 2024-04-25T14:03:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.560+0000 7f5404ff9640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 0x7f53f40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.560+0000 7f5407fff640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 0x7f53f40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.560+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 5853+0+0 (secure 0 0 0) 0x7f54000b5b80 con 0x7f5408163760 2024-04-25T14:03:15.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.560+0000 7f5407fff640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 0x7f53f40769f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f540808d310 tx=0x7f53f8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:15.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.564+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f540004b020 con 0x7f5408163760 2024-04-25T14:03:15.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:15.743+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5408161190 con 0x7f5408163760 2024-04-25T14:03:16.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.415+0000 7f5404ff9640 1 -- 172.21.15.150:0/2400154322 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v30) ==== 178+0+0 (secure 0 0 0) 0x7f5400080280 con 0x7f5408163760 2024-04-25T14:03:16.416 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data_ec' removed 2024-04-25T14:03:16.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.418+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 msgr2=0x7f53f40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.418+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 0x7f53f40769f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f540808d310 tx=0x7f53f8008040 comp rx=0 tx=0).stop 2024-04-25T14:03:16.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 msgr2=0x7f5408149d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f540002ac50 tx=0x7f5400004a40 comp rx=0 tx=0).stop 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 shutdown_connections 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f53f40744e0 0x7f53f40769f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5408163760 0x7f5408149d40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f54080d13f0 0x7f5408149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 --2- 172.21.15.150:0/2400154322 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f540806d4c0 0x7f540814d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 >> 172.21.15.150:0/2400154322 conn(0x7f54080d2720 msgr2=0x7f5408161c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 shutdown_connections 2024-04-25T14:03:16.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.419+0000 7f540e372640 1 -- 172.21.15.150:0/2400154322 wait complete. 2024-04-25T14:03:16.436 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T14:03:16.436 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T14:03:16.436 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T14:03:16.436 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:16.436 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T14:03:16.436 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T14:03:16.437 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T14:03:16.437 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:16.437 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T14:03:16.437 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T14:03:16.437 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T14:03:16.437 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:16.437 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T14:03:16.438 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T14:03:16.438 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T14:03:16.438 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:16.447 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T14:03:16.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T14:03:16.448 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T14:03:16.448 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T14:03:16.449 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T14:03:16.450 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T14:03:16.451 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T14:03:16.452 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T14:03:16.489 INFO:tasks.ceph.mds.a:Started 2024-04-25T14:03:16.489 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:16.490 INFO:tasks.ceph.mds.c:Started 2024-04-25T14:03:16.490 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:16.493 INFO:tasks.ceph.mds.d:Started 2024-04-25T14:03:16.494 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:16.495 INFO:tasks.ceph.mds.b:Started 2024-04-25T14:03:16.495 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:16.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T14:03:16.579 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:03:16.579 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-04-25T14:03:16.586 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:03:16.586 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:03:16.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.614+0000 7f07e0f0d640 1 Processor -- start 2024-04-25T14:03:16.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.614+0000 7f07e0f0d640 1 -- start start 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.615+0000 7f07e0f0d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.615+0000 7f07e0f0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 0x7f07dc15e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.615+0000 7f07e0f0d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07dc15f330 con 0x7f07dc0b54a0 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.615+0000 7f07e0f0d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f07dc162350 con 0x7f07dc15e590 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.615+0000 7f07e0f0d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc161d00 con 0x7f07dc1627e0 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07dad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 0x7f07dc15e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:52364/0 (socket says 172.21.15.150:52364) 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07dad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 0x7f07dc15e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60232/0 (socket says 172.21.15.150:60232) 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 -- 172.21.15.150:0/2570148625 learned_addr learned my addr 172.21.15.150:0/2570148625 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 -- 172.21.15.150:0/2570148625 >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 msgr2=0x7f07dc15e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 --2- 172.21.15.150:0/2570148625 >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 0x7f07dc15e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 -- 172.21.15.150:0/2570148625 >> v1:172.21.15.187:6789/0 conn(0x7f07dc0b54a0 legacy=0x7f07dc0b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07da59c640 1 -- 172.21.15.150:0/2570148625 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc161fe0 con 0x7f07dc1627e0 2024-04-25T14:03:16.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.616+0000 7f07d9d9b640 1 -- 172.21.15.150:0/2570148625 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683617303 0 0) 0x7f07dc15f330 con 0x7f07dc0b54a0 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.617+0000 7f07da59c640 1 --2- 172.21.15.150:0/2570148625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009870 tx=0x7f07cc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=923efe20024236ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.617+0000 7f07d9d9b640 1 -- 172.21.15.150:0/2570148625 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc037070 con 0x7f07dc1627e0 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.617+0000 7f07d9d9b640 1 -- 172.21.15.150:0/2570148625 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07cc032440 con 0x7f07dc1627e0 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.617+0000 7f07d9d9b640 1 -- 172.21.15.150:0/2570148625 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07cc032810 con 0x7f07dc1627e0 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 -- 172.21.15.150:0/2570148625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 msgr2=0x7f07dc15bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/2570148625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07cc009870 tx=0x7f07cc02a270 comp rx=0 tx=0).stop 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 -- 172.21.15.150:0/2570148625 shutdown_connections 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/2570148625 >> v2:172.21.15.150:3300/0 conn(0x7f07dc15e590 0x7f07dc15e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/2570148625 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc1627e0 0x7f07dc15bbf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 -- 172.21.15.150:0/2570148625 >> 172.21.15.150:0/2570148625 conn(0x7f07dc0d04a0 msgr2=0x7f07dc0d08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 -- 172.21.15.150:0/2570148625 shutdown_connections 2024-04-25T14:03:16.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.618+0000 7f07e0f0d640 1 -- 172.21.15.150:0/2570148625 wait complete. 2024-04-25T14:03:16.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.620+0000 7f07e0f0d640 1 Processor -- start 2024-04-25T14:03:16.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.620+0000 7f07e0f0d640 1 -- start start 2024-04-25T14:03:16.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc0b54a0 0x7f07dc143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc1627e0 0x7f07dc146350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b2640 con 0x7f07dc15e590 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f07dc0b24c0 con 0x7f07dc1627e0 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07e0f0d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f07dc0b27c0 con 0x7f07dc0b54a0 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60234/0 (socket says 172.21.15.150:60234) 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07d3fff640 1 -- 172.21.15.150:0/1177179244 learned_addr learned my addr 172.21.15.150:0/1177179244 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07da59c640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc0b54a0 0x7f07dc143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.621+0000 7f07dad9d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc1627e0 0x7f07dc146350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 -- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc0b54a0 msgr2=0x7f07dc143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc0b54a0 0x7f07dc143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 -- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc1627e0 msgr2=0x7f07dc146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc1627e0 0x7f07dc146350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 -- 172.21.15.150:0/1177179244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07dc0b5250 con 0x7f07dc15e590 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d3fff640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c3f0 tx=0x7f07c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c4017070 con 0x7f07dc15e590 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07c40035f0 con 0x7f07dc15e590 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07dc1547a0 con 0x7f07dc15e590 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07c40039e0 con 0x7f07dc15e590 2024-04-25T14:03:16.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.622+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07dc154d60 con 0x7f07dc15e590 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.623+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07dc076a40 con 0x7f07dc15e590 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.624+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f07c401a070 con 0x7f07dc15e590 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.624+0000 7f07d2ffd640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 0x7f07bc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.624+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5335+0+0 (secure 0 0 0) 0x7f07c40943c0 con 0x7f07dc15e590 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.624+0000 7f07da59c640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 0x7f07bc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:16.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.624+0000 7f07da59c640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 0x7f07bc0769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f07cc034000 tx=0x7f07cc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:16.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.629+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f07c405e670 con 0x7f07dc15e590 2024-04-25T14:03:16.818 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:16.817+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f07dc0b4fe0 con 0x7f07dc15e590 2024-04-25T14:03:17.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.425+0000 7f07d2ffd640 1 -- 172.21.15.150:0/1177179244 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v32) ==== 97+0+0 (secure 0 0 0) 0x7f07c4062970 con 0x7f07dc15e590 2024-04-25T14:03:17.425 INFO:teuthology.orchestra.run.smithi150.stderr: removed all blocklist entries 2024-04-25T14:03:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 msgr2=0x7f07bc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 0x7f07bc0769c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f07cc034000 tx=0x7f07cc02e000 comp rx=0 tx=0).stop 2024-04-25T14:03:17.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 msgr2=0x7f07dc0b4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f07c400c3f0 tx=0x7f07c400c8b0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 shutdown_connections 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f07bc0744b0 0x7f07bc0769c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f07dc1627e0 0x7f07dc146350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f07dc15e590 0x7f07dc0b4600 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 --2- 172.21.15.150:0/1177179244 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f07dc0b54a0 0x7f07dc143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 >> 172.21.15.150:0/1177179244 conn(0x7f07dc0d04a0 msgr2=0x7f07dc0d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 shutdown_connections 2024-04-25T14:03:17.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.428+0000 7f07e0f0d640 1 -- 172.21.15.150:0/1177179244 wait complete. 2024-04-25T14:03:17.441 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T14:03:17.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.549+0000 7fbc27bc7640 1 Processor -- start 2024-04-25T14:03:17.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.549+0000 7fbc27bc7640 1 -- start start 2024-04-25T14:03:17.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc27bc7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc201601d0 0x7fbc200d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc27bc7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 0x7fbc2006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc27bc7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc2006c570 con 0x7fbc2006d830 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc27bc7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbc201605e0 con 0x7fbc201601d0 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc27bc7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbc200d5980 con 0x7fbc200b9070 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc201601d0 0x7fbc200d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc201601d0 0x7fbc200d5100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39950/0 (socket says 172.21.15.150:39950) 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 -- 172.21.15.150:0/2204585417 learned_addr learned my addr 172.21.15.150:0/2204585417 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25162640 1 --2- 172.21.15.150:0/2204585417 >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 0x7fbc2006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 -- 172.21.15.150:0/2204585417 >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 msgr2=0x7fbc2006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 --2- 172.21.15.150:0/2204585417 >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 0x7fbc2006d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 -- 172.21.15.150:0/2204585417 >> v1:172.21.15.187:6789/0 conn(0x7fbc2006d830 legacy=0x7fbc200d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25963640 1 -- 172.21.15.150:0/2204585417 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2006c810 con 0x7fbc201601d0 2024-04-25T14:03:17.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.551+0000 7fbc25162640 1 --2- 172.21.15.150:0/2204585417 >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 0x7fbc2006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.552+0000 7fbc25963640 1 --2- 172.21.15.150:0/2204585417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc200d5100 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbc200d3ce0 tx=0x7fbc1c02a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ff69eb430991d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.552+0000 7fbc24961640 1 -- 172.21.15.150:0/2204585417 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c036070 con 0x7fbc201601d0 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.552+0000 7fbc24961640 1 -- 172.21.15.150:0/2204585417 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c031600 con 0x7fbc201601d0 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.552+0000 7fbc24961640 1 -- 172.21.15.150:0/2204585417 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0319d0 con 0x7fbc201601d0 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2204585417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 msgr2=0x7fbc200d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2204585417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc200d5100 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbc200d3ce0 tx=0x7fbc1c02a640 comp rx=0 tx=0).stop 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2204585417 shutdown_connections 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2204585417 >> v2:172.21.15.187:3300/0 conn(0x7fbc200b9070 0x7fbc2006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2204585417 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc200d5100 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2204585417 >> 172.21.15.150:0/2204585417 conn(0x7fbc200d5d90 msgr2=0x7fbc200d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:17.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2204585417 shutdown_connections 2024-04-25T14:03:17.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.553+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2204585417 wait complete. 2024-04-25T14:03:17.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.555+0000 7fbc27bc7640 1 Processor -- start 2024-04-25T14:03:17.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.555+0000 7fbc27bc7640 1 -- start start 2024-04-25T14:03:17.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc2006d830 0x7fbc201523f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 0x7fbc201ea520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbc200d5640 con 0x7fbc201601d0 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2006cdd0 con 0x7fbc200b9070 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc27bc7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc200b9480 con 0x7fbc2006d830 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39960/0 (socket says 172.21.15.150:39960) 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 -- 172.21.15.150:0/2259025488 learned_addr learned my addr 172.21.15.150:0/2259025488 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc25162640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 0x7fbc201ea520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc25963640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc2006d830 0x7fbc201523f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 -- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc2006d830 msgr2=0x7fbc201523f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc2006d830 0x7fbc201523f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 -- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 msgr2=0x7fbc201ea520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 0x7fbc201ea520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 -- 172.21.15.150:0/2259025488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc200cae20 con 0x7fbc201601d0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc25162640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 0x7fbc201ea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.556+0000 7fbc26164640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b410 tx=0x7fbc1800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.557+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18015030 con 0x7fbc201601d0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.557+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc18004020 con 0x7fbc201601d0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.557+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc200cb100 con 0x7fbc201601d0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.557+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc18004410 con 0x7fbc201601d0 2024-04-25T14:03:17.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.557+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc20159bd0 con 0x7fbc201601d0 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.558+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc20159e30 con 0x7fbc201601d0 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.558+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbc18018070 con 0x7fbc201601d0 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.559+0000 7fbc167fc640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 0x7fbc040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.559+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fbc18094a60 con 0x7fbc201601d0 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.559+0000 7fbc25963640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 0x7fbc040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.559+0000 7fbc25963640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 0x7fbc040769c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c033000 tx=0x7fbc1c009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.564+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc1805f440 con 0x7fbc201601d0 2024-04-25T14:03:17.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.803+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fbc201eaf00 con 0x7fbc201601d0 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.804+0000 7fbc167fc640 1 -- 172.21.15.150:0/2259025488 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fbc1805ed40 con 0x7fbc201601d0 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_dump": [ 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.a", 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmu4BmJxAA97zMMh3D3KUNtsDbGwkzDA==", 2024-04-25T14:03:17.804 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.b", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmwLnMMxAATYxzVsSAZZUjQJuFzeuaSA==", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:03:17.805 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.c", 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmGRrILhAAIg2pUAKfYtkPHonZ2jFm2Q==", 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.806 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.d", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmwO4tOxAA8v8nYbz09RgqeKeG1kP8Aw==", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.0", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAqYipma5wlGhAA4H2WY5d5faQLkw54oTdhvg==", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.807 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.1", 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAtYipm9+3RGBAANzSmNtvSkPp3PZqNPOc5hQ==", 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.808 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.2", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAwYipmb72gFxAAXszjhp98sXfU3mERSpXVYA==", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.3", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAzYipmN9yoFhAAUeT/1IDgSofY9Txb6wBIoA==", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.809 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.4", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA5YipmorY9HhAAStUVEPsdQwhrk3CRCyKReg==", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.5", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA8YipmFsVgHRAAuxUDVhP1YYbOIuqKnk/0rg==", 2024-04-25T14:03:17.810 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.6", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA/YipmCeXsGxAAMKhPXK0qkbkSvZjER5hCxA==", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.811 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.7", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBCYipmfInFGhAAQzv1lzLJbEZsVlzxNER6xw==", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.0", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ==", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.812 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.1", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmDUJACxAAcFyylAXclzW4D6C20RFk4Q==", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.813 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.2", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipm2NSoBxAABm1fUKNHFMNuiHMmSMrKlA==", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.3", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmm2fQDhAANojc8uxJB15T5dVpgYA4Vg==", 2024-04-25T14:03:17.814 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.admin", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmNaqiCRAANv+ub/K0QsBwYm9UTT+2Fg==", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.815 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-mds", 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipm03NcGBAA5jgwAp3vYucqYDYc1e//Dw==", 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-mds" 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-mgr", 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipmVXxcGBAA6pE+LbzVcaHEzZwaOlwc3Q==", 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-25T14:03:17.816 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-osd", 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipmWYRcGBAAFn7HRZR8WG1Lqu7rwf5tgQ==", 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-osd" 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rbd", 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipmfotcGBAAzuIux+wElmjasuRrzCfGiA==", 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-25T14:03:17.817 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipmupJcGBAAR1m5y9A96Svdl2Tgcq/0Vg==", 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.bootstrap-rgw", 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBIYipmC5pcGBAAyya1bB7Q21BRaZWGJweDWQ==", 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-25T14:03:17.818 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.x", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmoMzCIxAAMQ8RkeGQMgn4UuygerC8eg==", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.y", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmdMg8HhAAGiDa9V7HIfyPF1HvP9u4rQ==", 2024-04-25T14:03:17.819 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-04-25T14:03:17.820 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:03:17.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 msgr2=0x7fbc040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 0x7fbc040769c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c033000 tx=0x7fbc1c009000 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 msgr2=0x7fbc20161de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fbc1800b410 tx=0x7fbc1800b8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 shutdown_connections 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbc040744b0 0x7fbc040769c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc201601d0 0x7fbc20161de0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc200b9070 0x7fbc201ea520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 --2- 172.21.15.150:0/2259025488 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc2006d830 0x7fbc201523f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 >> 172.21.15.150:0/2259025488 conn(0x7fbc200d5d90 msgr2=0x7fbc200ba9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 shutdown_connections 2024-04-25T14:03:17.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.806+0000 7fbc27bc7640 1 -- 172.21.15.150:0/2259025488 wait complete. 2024-04-25T14:03:17.823 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-25T14:03:17.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.931+0000 7f65b45e3640 1 Processor -- start 2024-04-25T14:03:17.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.931+0000 7f65b45e3640 1 -- start start 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b45e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac15dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b45e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65ac162540 0x7f65ac0cca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b45e3640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65ac0cd380 con 0x7f65ac0ccf70 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b45e3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f65ac06deb0 con 0x7f65ac162540 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b45e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f65ac06d610 con 0x7f65ac06daa0 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b1b7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65ac162540 0x7f65ac0cca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.932+0000 7f65b1b7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f65ac162540 0x7f65ac0cca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39970/0 (socket says 172.21.15.150:39970) 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 -- 172.21.15.150:0/1294191619 learned_addr learned my addr 172.21.15.150:0/1294191619 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b237f640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac15dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 -- 172.21.15.150:0/1294191619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 msgr2=0x7f65ac15dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac15dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 -- 172.21.15.150:0/1294191619 >> v1:172.21.15.187:6789/0 conn(0x7f65ac0ccf70 legacy=0x7f65ac0d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 -- 172.21.15.150:0/1294191619 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac0cc5f0 con 0x7f65ac162540 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.933+0000 7f65b1b7e640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac0cca30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f659c009870 tx=0x7f659c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89582de4d2fb89f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b137d640 1 -- 172.21.15.150:0/1294191619 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c037070 con 0x7f65ac162540 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b137d640 1 -- 172.21.15.150:0/1294191619 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f659c032440 con 0x7f65ac162540 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b137d640 1 -- 172.21.15.150:0/1294191619 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c032810 con 0x7f65ac162540 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 -- 172.21.15.150:0/1294191619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 msgr2=0x7f65ac0cca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac0cca30 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f659c009870 tx=0x7f659c02a270 comp rx=0 tx=0).stop 2024-04-25T14:03:17.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 -- 172.21.15.150:0/1294191619 shutdown_connections 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac0cca30 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 --2- 172.21.15.150:0/1294191619 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac15dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 -- 172.21.15.150:0/1294191619 >> 172.21.15.150:0/1294191619 conn(0x7f65ac0ce390 msgr2=0x7f65ac0ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.934+0000 7f65b45e3640 1 -- 172.21.15.150:0/1294191619 shutdown_connections 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.935+0000 7f65b45e3640 1 -- 172.21.15.150:0/1294191619 wait complete. 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.936+0000 7f65b45e3640 1 Processor -- start 2024-04-25T14:03:17.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.936+0000 7f65b45e3640 1 -- start start 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac14dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65ac0ccf70 0x7f65ac14d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac06cfc0 con 0x7f65ac162540 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac06cce0 con 0x7f65ac0ccf70 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f65ac06d2a0 con 0x7f65ac06daa0 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39974/0 (socket says 172.21.15.150:39974) 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 -- 172.21.15.150:0/730164269 learned_addr learned my addr 172.21.15.150:0/730164269 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b237f640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac14dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b1b7e640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65ac0ccf70 0x7f65ac14d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 -- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 msgr2=0x7f65ac14dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac14dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 -- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65ac0ccf70 msgr2=0x7f65ac14d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65ac0ccf70 0x7f65ac14d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 -- 172.21.15.150:0/730164269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac14d980 con 0x7f65ac162540 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b2b80640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f65a400c6e0 tx=0x7f65a400cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4003b70 con 0x7f65ac162540 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a4003d30 con 0x7f65ac162540 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac14af90 con 0x7f65ac162540 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.938+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4005310 con 0x7f65ac162540 2024-04-25T14:03:17.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.939+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac14b550 con 0x7f65ac162540 2024-04-25T14:03:17.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.940+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac076a40 con 0x7f65ac162540 2024-04-25T14:03:17.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.940+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f65a40054b0 con 0x7f65ac162540 2024-04-25T14:03:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.940+0000 7f65a2ffd640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 0x7f65900769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.941+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f65a4094d90 con 0x7f65ac162540 2024-04-25T14:03:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.941+0000 7f65b237f640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 0x7f65900769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:17.942 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.941+0000 7f65b237f640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 0x7f65900769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65ac09fb30 tx=0x7f65a8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:17.945 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:17.945+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f65ac0cd380 con 0x7f65ac162540 2024-04-25T14:03:18.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.189+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f65ac0cfe10 con 0x7f65ac162540 2024-04-25T14:03:18.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.191+0000 7f65a2ffd640 1 -- 172.21.15.150:0/730164269 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f65a405f090 con 0x7f65ac162540 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 msgr2=0x7f65900769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 0x7f65900769c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65ac09fb30 tx=0x7f65a8009190 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 msgr2=0x7f65ac09dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f65a400c6e0 tx=0x7f65a400cba0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 shutdown_connections 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f65900744b0 0x7f65900769c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f65ac162540 0x7f65ac09dad0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f65ac0ccf70 0x7f65ac14d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 --2- 172.21.15.150:0/730164269 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f65ac06daa0 0x7f65ac14dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.194 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.193+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 >> 172.21.15.150:0/730164269 conn(0x7f65ac0ce390 msgr2=0x7f65ac0b5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:18.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.194+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 shutdown_connections 2024-04-25T14:03:18.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.194+0000 7f65b45e3640 1 -- 172.21.15.150:0/730164269 wait complete. 2024-04-25T14:03:18.205 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-25T14:03:18.313 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.313+0000 7f89d9ab6640 1 Processor -- start 2024-04-25T14:03:18.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.313+0000 7f89d9ab6640 1 -- start start 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d9ab6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f89d40d3280 0x7f89d4160360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d9ab6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f89d40735b0 0x7f89d40b8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d9ab6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89d4164250 con 0x7f89d40b6800 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d9ab6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f89d41644f0 con 0x7f89d40735b0 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d9ab6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f89d41647d0 con 0x7f89d40d3280 2024-04-25T14:03:18.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f89d40735b0 0x7f89d40b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d2ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f89d40b6800 0x7f89d4072ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39040/0 (socket says 172.21.15.150:39040) 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.314+0000 7f89d2ffd640 1 -- 172.21.15.150:0/2532999809 learned_addr learned my addr 172.21.15.150:0/2532999809 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d37fe640 1 --2- 172.21.15.150:0/2532999809 >> v2:172.21.15.187:3300/0 conn(0x7f89d40d3280 0x7f89d4160360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d3fff640 1 -- 172.21.15.150:0/2532999809 >> v2:172.21.15.187:3300/0 conn(0x7f89d40d3280 msgr2=0x7f89d4160360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d3fff640 1 --2- 172.21.15.150:0/2532999809 >> v2:172.21.15.187:3300/0 conn(0x7f89d40d3280 0x7f89d4160360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d3fff640 1 -- 172.21.15.150:0/2532999809 >> v1:172.21.15.150:6789/0 conn(0x7f89d40b6800 legacy=0x7f89d4072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d3fff640 1 -- 172.21.15.150:0/2532999809 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d4164ab0 con 0x7f89d40735b0 2024-04-25T14:03:18.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d27fc640 1 -- 172.21.15.150:0/2532999809 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 143933878 0 0) 0x7f89d4164250 con 0x7f89d40b6800 2024-04-25T14:03:18.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.315+0000 7f89d3fff640 1 --2- 172.21.15.150:0/2532999809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40735b0 0x7f89d40b8460 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f89d40d4990 tx=0x7f89cc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1f4ac3264db05f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d27fc640 1 -- 172.21.15.150:0/2532999809 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc007740 con 0x7f89d40735b0 2024-04-25T14:03:18.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d27fc640 1 -- 172.21.15.150:0/2532999809 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc0078e0 con 0x7f89d40735b0 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d27fc640 1 -- 172.21.15.150:0/2532999809 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc007cb0 con 0x7f89d40735b0 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 -- 172.21.15.150:0/2532999809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40735b0 msgr2=0x7f89d40b8460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/2532999809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40735b0 0x7f89d40b8460 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f89d40d4990 tx=0x7f89cc02a680 comp rx=0 tx=0).stop 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 -- 172.21.15.150:0/2532999809 shutdown_connections 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/2532999809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40735b0 0x7f89d40b8460 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/2532999809 >> v2:172.21.15.187:3300/0 conn(0x7f89d40d3280 0x7f89d4160360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 -- 172.21.15.150:0/2532999809 >> 172.21.15.150:0/2532999809 conn(0x7f89d40d39b0 msgr2=0x7f89d40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 -- 172.21.15.150:0/2532999809 shutdown_connections 2024-04-25T14:03:18.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.316+0000 7f89d9ab6640 1 -- 172.21.15.150:0/2532999809 wait complete. 2024-04-25T14:03:18.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.318+0000 7f89d9ab6640 1 Processor -- start 2024-04-25T14:03:18.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.318+0000 7f89d9ab6640 1 -- start start 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b6800 0x7f89d4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40d3280 0x7f89d40c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f89d40ce000 con 0x7f89d40c7750 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f89d4163f40 con 0x7f89d40d3280 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f89d40b89a0 con 0x7f89d40b6800 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39984/0 (socket says 172.21.15.150:39984) 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 -- 172.21.15.150:0/3194906092 learned_addr learned my addr 172.21.15.150:0/3194906092 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d2ffd640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40d3280 0x7f89d40c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d37fe640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b6800 0x7f89d4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 -- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b6800 msgr2=0x7f89d4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b6800 0x7f89d4159e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 -- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40d3280 msgr2=0x7f89d40c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40d3280 0x7f89d40c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 -- 172.21.15.150:0/3194906092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89d415a3d0 con 0x7f89d40c7750 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d3fff640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009e50 tx=0x7f89cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc03f070 con 0x7f89d40c7750 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89cc031bf0 con 0x7f89d40c7750 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.320+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89d40c87a0 con 0x7f89d40c7750 2024-04-25T14:03:18.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.321+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89cc044460 con 0x7f89d40c7750 2024-04-25T14:03:18.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.321+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89d40c81e0 con 0x7f89d40c7750 2024-04-25T14:03:18.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.322+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89d4076a40 con 0x7f89d40c7750 2024-04-25T14:03:18.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.322+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f89cc007790 con 0x7f89d40c7750 2024-04-25T14:03:18.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.323+0000 7f89bbfff640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 0x7f89c00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.323+0000 7f89d37fe640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 0x7f89c00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.323+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f89cc0b6930 con 0x7f89d40c7750 2024-04-25T14:03:18.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.323+0000 7f89d37fe640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 0x7f89c00769c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006fd0 tx=0x7f89c4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.328+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f89cc081280 con 0x7f89d40c7750 2024-04-25T14:03:18.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.577+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f89d40d3060 con 0x7f89d40c7750 2024-04-25T14:03:18.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.579+0000 7f89bbfff640 1 -- 172.21.15.150:0/3194906092 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f89cc089300 con 0x7f89d40c7750 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 msgr2=0x7f89c00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 0x7f89c00769c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f89c4006fd0 tx=0x7f89c4008040 comp rx=0 tx=0).stop 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 msgr2=0x7f89d40ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009e50 tx=0x7f89cc004a40 comp rx=0 tx=0).stop 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 shutdown_connections 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f89c00744b0 0x7f89c00769c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f89d40c7750 0x7f89d40ce7b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f89d40d3280 0x7f89d40c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 --2- 172.21.15.150:0/3194906092 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f89d40b6800 0x7f89d4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.581+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 >> 172.21.15.150:0/3194906092 conn(0x7f89d40d39b0 msgr2=0x7f89d415d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:18.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.582+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 shutdown_connections 2024-04-25T14:03:18.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.582+0000 7f89d9ab6640 1 -- 172.21.15.150:0/3194906092 wait complete. 2024-04-25T14:03:18.593 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-25T14:03:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.701+0000 7f4feff9d640 1 Processor -- start 2024-04-25T14:03:18.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.702+0000 7f4feff9d640 1 -- start start 2024-04-25T14:03:18.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4feff9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fe815a850 0x7f4fe809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4feff9d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 0x7f4fe80b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4feff9d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4fe815ac60 con 0x7f4fe8160770 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4feff9d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4fe809bdf0 con 0x7f4fe815a850 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4feff9d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe8160ce0 con 0x7f4fe809b9e0 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fe815a850 0x7f4fe809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4fe815a850 0x7f4fe809b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:39998/0 (socket says 172.21.15.150:39998) 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fedd39640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4fe8160770 0x7f4fe806f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39052/0 (socket says 172.21.15.150:39052) 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 -- 172.21.15.150:0/4050799985 learned_addr learned my addr 172.21.15.150:0/4050799985 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fee53a640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 0x7f4fe80b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 -- 172.21.15.150:0/4050799985 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 msgr2=0x7f4fe80b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 0x7f4fe80b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 -- 172.21.15.150:0/4050799985 >> v1:172.21.15.150:6789/0 conn(0x7f4fe8160770 legacy=0x7f4fe806f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fed538640 1 -- 172.21.15.150:0/4050799985 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe8160fc0 con 0x7f4fe815a850 2024-04-25T14:03:18.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.703+0000 7f4fee53a640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 0x7f4fe80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.704+0000 7f4fed538640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe809b4a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009870 tx=0x7f4fd802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84d958229eef7549 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.704+0000 7f4fecd37640 1 -- 172.21.15.150:0/4050799985 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8037070 con 0x7f4fe815a850 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.704+0000 7f4fecd37640 1 -- 172.21.15.150:0/4050799985 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd8032440 con 0x7f4fe815a850 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.704+0000 7f4fecd37640 1 -- 172.21.15.150:0/4050799985 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8032810 con 0x7f4fe815a850 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 -- 172.21.15.150:0/4050799985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 msgr2=0x7f4fe809b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe809b4a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009870 tx=0x7f4fd802a270 comp rx=0 tx=0).stop 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 -- 172.21.15.150:0/4050799985 shutdown_connections 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe809b9e0 0x7f4fe80b3070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 --2- 172.21.15.150:0/4050799985 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe809b4a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 -- 172.21.15.150:0/4050799985 >> 172.21.15.150:0/4050799985 conn(0x7f4fe80b7680 msgr2=0x7f4fe80b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 -- 172.21.15.150:0/4050799985 shutdown_connections 2024-04-25T14:03:18.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.705+0000 7f4feff9d640 1 -- 172.21.15.150:0/4050799985 wait complete. 2024-04-25T14:03:18.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.707+0000 7f4feff9d640 1 Processor -- start 2024-04-25T14:03:18.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.707+0000 7f4feff9d640 1 -- start start 2024-04-25T14:03:18.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe809b9e0 0x7f4fe8156070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe8160770 0x7f4fe81588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe80b35b0 con 0x7f4fe815a850 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe8161580 con 0x7f4fe809b9e0 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4feff9d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4fe80b3730 con 0x7f4fe8160770 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40012/0 (socket says 172.21.15.150:40012) 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 -- 172.21.15.150:0/1029677460 learned_addr learned my addr 172.21.15.150:0/1029677460 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fee53a640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe8160770 0x7f4fe81588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fedd39640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe809b9e0 0x7f4fe8156070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 -- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe8160770 msgr2=0x7f4fe81588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe8160770 0x7f4fe81588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 -- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe809b9e0 msgr2=0x7f4fe8156070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe809b9e0 0x7f4fe8156070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.708+0000 7f4fed538640 1 -- 172.21.15.150:0/1029677460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe81f78f0 con 0x7f4fe815a850 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4fed538640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8000c00 tx=0x7f4fd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8037040 con 0x7f4fe815a850 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fd8042070 con 0x7f4fe815a850 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe81565b0 con 0x7f4fe815a850 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8030bd0 con 0x7f4fe815a850 2024-04-25T14:03:18.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.709+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe80d3380 con 0x7f4fe815a850 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.710+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fe80b11d0 con 0x7f4fe815a850 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.710+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4fd8047460 con 0x7f4fe815a850 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.711+0000 7f4fde7fc640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 0x7f4fcc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.711+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f4fd80b5c80 con 0x7f4fe815a850 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.711+0000 7f4fedd39640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 0x7f4fcc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:18.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.711+0000 7f4fedd39640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 0x7f4fcc0769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40012e0 tx=0x7f4fe4005b70 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:18.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.715+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fe80b11d0 con 0x7f4fe815a850 2024-04-25T14:03:18.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.958+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f4fe8076a40 con 0x7f4fe815a850 2024-04-25T14:03:18.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.960+0000 7f4fde7fc640 1 -- 172.21.15.150:0/1029677460 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f4fd8080010 con 0x7f4fe815a850 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 msgr2=0x7f4fcc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 0x7f4fcc0769c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4fe40012e0 tx=0x7f4fe4005b70 comp rx=0 tx=0).stop 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 msgr2=0x7f4fe815b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8000c00 tx=0x7f4fd80046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 shutdown_connections 2024-04-25T14:03:18.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4fcc0744b0 0x7f4fcc0769c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4fe8160770 0x7f4fe81588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4fe815a850 0x7f4fe815b690 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 --2- 172.21.15.150:0/1029677460 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4fe809b9e0 0x7f4fe8156070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 >> 172.21.15.150:0/1029677460 conn(0x7f4fe80b7680 msgr2=0x7f4fe80684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 shutdown_connections 2024-04-25T14:03:18.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:18.963+0000 7f4feff9d640 1 -- 172.21.15.150:0/1029677460 wait complete. 2024-04-25T14:03:18.974 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-25T14:03:19.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.083+0000 7f56fd3d6640 1 Processor -- start 2024-04-25T14:03:19.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.083+0000 7f56fd3d6640 1 -- start start 2024-04-25T14:03:19.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56fd3d6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56f8160150 0x7f56f80b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56fd3d6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f56f806d800 0x7f56f80d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56fd3d6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f80d4f20 con 0x7f56f80b4c10 2024-04-25T14:03:19.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56fd3d6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f56f806cc60 con 0x7f56f8160150 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56fd3d6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f56f8160560 con 0x7f56f806d800 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f56f8160150 0x7f56f80b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f56f80b4c10 0x7f56f806d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39058/0 (socket says 172.21.15.150:39058) 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f6ffd640 1 -- 172.21.15.150:0/1264206012 learned_addr learned my addr 172.21.15.150:0/1264206012 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f77fe640 1 --2- 172.21.15.150:0/1264206012 >> v2:172.21.15.187:3300/0 conn(0x7f56f806d800 0x7f56f80d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f67fc640 1 -- 172.21.15.150:0/1264206012 >> v2:172.21.15.187:3300/0 conn(0x7f56f806d800 msgr2=0x7f56f80d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f67fc640 1 --2- 172.21.15.150:0/1264206012 >> v2:172.21.15.187:3300/0 conn(0x7f56f806d800 0x7f56f80d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f67fc640 1 -- 172.21.15.150:0/1264206012 >> v1:172.21.15.150:6789/0 conn(0x7f56f80b4c10 legacy=0x7f56f806d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f5ffb640 1 -- 172.21.15.150:0/1264206012 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780719939 0 0) 0x7f56f80d4f20 con 0x7f56f80b4c10 2024-04-25T14:03:19.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.085+0000 7f56f67fc640 1 -- 172.21.15.150:0/1264206012 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f80d5200 con 0x7f56f8160150 2024-04-25T14:03:19.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.086+0000 7f56f67fc640 1 --2- 172.21.15.150:0/1264206012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f80b91f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f56f80d0690 tx=0x7f56ec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373fe38829e75aee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.086+0000 7f56f5ffb640 1 -- 172.21.15.150:0/1264206012 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec037070 con 0x7f56f8160150 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.086+0000 7f56f5ffb640 1 -- 172.21.15.150:0/1264206012 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56ec032440 con 0x7f56f8160150 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.086+0000 7f56f5ffb640 1 -- 172.21.15.150:0/1264206012 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56ec032810 con 0x7f56f8160150 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1264206012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 msgr2=0x7f56f80b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1264206012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f80b91f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f56f80d0690 tx=0x7f56ec02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1264206012 shutdown_connections 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1264206012 >> v2:172.21.15.187:3300/0 conn(0x7f56f806d800 0x7f56f80d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1264206012 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f80b91f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1264206012 >> 172.21.15.150:0/1264206012 conn(0x7f56f80b7690 msgr2=0x7f56f80b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1264206012 shutdown_connections 2024-04-25T14:03:19.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.087+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1264206012 wait complete. 2024-04-25T14:03:19.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.089+0000 7f56fd3d6640 1 Processor -- start 2024-04-25T14:03:19.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.089+0000 7f56fd3d6640 1 -- start start 2024-04-25T14:03:19.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56f806d800 0x7f56f8160710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56f80b4c10 0x7f56f80ad000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f56f806c8f0 con 0x7f56f8160150 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f56f806c610 con 0x7f56f806d800 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56fd3d6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f56f806dc10 con 0x7f56f80b4c10 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40032/0 (socket says 172.21.15.150:40032) 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 -- 172.21.15.150:0/1725986281 learned_addr learned my addr 172.21.15.150:0/1725986281 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f6ffd640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56f806d800 0x7f56f8160710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f67fc640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56f80b4c10 0x7f56f80ad000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 -- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56f80b4c10 msgr2=0x7f56f80ad000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56f80b4c10 0x7f56f80ad000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 -- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56f806d800 msgr2=0x7f56f8160710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56f806d800 0x7f56f8160710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.090+0000 7f56f77fe640 1 -- 172.21.15.150:0/1725986281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f815acc0 con 0x7f56f8160150 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56f77fe640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f56f000d5a0 tx=0x7f56f000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f00125f0 con 0x7f56f8160150 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56f0019070 con 0x7f56f8160150 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56f81f6d60 con 0x7f56f8160150 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56f00129f0 con 0x7f56f8160150 2024-04-25T14:03:19.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.091+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56f80ac520 con 0x7f56f8160150 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.092+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f81f6ad0 con 0x7f56f8160150 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.093+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f56f0012b90 con 0x7f56f8160150 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.093+0000 7f56df7fe640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 0x7f56e40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.093+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f56f0095270 con 0x7f56f8160150 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.093+0000 7f56f6ffd640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 0x7f56e40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.094+0000 7f56f6ffd640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 0x7f56e40769f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56e80012e0 tx=0x7f56e8001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.098+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f56f005f580 con 0x7f56f8160150 2024-04-25T14:03:19.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.339+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f56f8076a40 con 0x7f56f8160150 2024-04-25T14:03:19.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.341+0000 7f56df7fe640 1 -- 172.21.15.150:0/1725986281 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f56f0063880 con 0x7f56f8160150 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 msgr2=0x7f56e40769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 0x7f56e40769f0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56e80012e0 tx=0x7f56e8001630 comp rx=0 tx=0).stop 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 msgr2=0x7f56f815a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f56f000d5a0 tx=0x7f56f000da60 comp rx=0 tx=0).stop 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 shutdown_connections 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f56e40744e0 0x7f56e40769f0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f56f8160150 0x7f56f815a1f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f56f80b4c10 0x7f56f80ad000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 --2- 172.21.15.150:0/1725986281 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f56f806d800 0x7f56f8160710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 >> 172.21.15.150:0/1725986281 conn(0x7f56f80b7690 msgr2=0x7f56f80d0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:19.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 shutdown_connections 2024-04-25T14:03:19.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.343+0000 7f56fd3d6640 1 -- 172.21.15.150:0/1725986281 wait complete. 2024-04-25T14:03:19.355 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-25T14:03:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.466+0000 7fd09b7e3640 1 Processor -- start 2024-04-25T14:03:19.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.467+0000 7fd09b7e3640 1 -- start start 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd09b7e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd0940b6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd09b7e3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd094164070 0x7fd094164480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd09b7e3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0941649c0 con 0x7fd0940d1db0 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd09b7e3640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd0940b9860 con 0x7fd094164070 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd09b7e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0940b92a0 con 0x7fd0940b67b0 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd099d80640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd0940d1db0 0x7fd0940d21c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39074/0 (socket says 172.21.15.150:39074) 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd099d80640 1 -- 172.21.15.150:0/1091112380 learned_addr learned my addr 172.21.15.150:0/1091112380 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:19.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 417626814 0 0) 0x7fd08c001110 con 0x7fd0940d1db0 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.468+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd080003520 con 0x7fd0940d1db0 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 --2- 172.21.15.150:0/1091112380 >> v2:172.21.15.187:3300/0 conn(0x7fd094164070 0x7fd094164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd09957f640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd0940b6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2667689404 0 0) 0x7fd080003520 con 0x7fd0940d1db0 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 msgr2=0x7fd0940b6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd0940b6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 >> v2:172.21.15.187:3300/0 conn(0x7fd094164070 msgr2=0x7fd094164480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 --2- 172.21.15.150:0/1091112380 >> v2:172.21.15.187:3300/0 conn(0x7fd094164070 0x7fd094164480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0940b9580 con 0x7fd0940d1db0 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fd08c001840 con 0x7fd0940d1db0 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 >> v1:172.21.15.150:6789/0 conn(0x7fd0940d1db0 legacy=0x7fd0940d21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800039b0 0x7fd080003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080004800 0x7fd080004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd080005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940b9860 con 0x7fd080004800 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940b92a0 con 0x7fd0800039b0 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd08000ba70 con 0x7fd0940b67b0 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080004800 0x7fd080004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd09957f640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800039b0 0x7fd080003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd099d80640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd080005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 -- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 msgr2=0x7fd080005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd0940b67b0 0x7fd080005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 -- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800039b0 msgr2=0x7fd080003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0800039b0 0x7fd080003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.469+0000 7fd098d7e640 1 -- 172.21.15.150:0/1091112380 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd080003520 con 0x7fd080004800 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.470+0000 7fd098d7e640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080004800 0x7fd080004c10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd084009a90 tx=0x7fd080005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549a36f95eb6ee00 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.470+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084036070 con 0x7fd080004800 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.470+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd084031400 con 0x7fd080004800 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.470+0000 7fd08bfff640 1 -- 172.21.15.150:0/1091112380 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084009690 con 0x7fd080004800 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 -- 172.21.15.150:0/1091112380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080004800 msgr2=0x7fd080004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/1091112380 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd080004800 0x7fd080004c10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd084009a90 tx=0x7fd080005830 comp rx=0 tx=0).stop 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd099d80640 1 -- 172.21.15.150:0/1091112380 reap_dead start 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 -- 172.21.15.150:0/1091112380 shutdown_connections 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 -- 172.21.15.150:0/1091112380 >> 172.21.15.150:0/1091112380 conn(0x7fd0940d3960 msgr2=0x7fd0940d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 -- 172.21.15.150:0/1091112380 shutdown_connections 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.471+0000 7fd09b7e3640 1 -- 172.21.15.150:0/1091112380 wait complete. 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.473+0000 7fd09b7e3640 1 Processor -- start 2024-04-25T14:03:19.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.473+0000 7fd09b7e3640 1 -- start start 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd080004800 0x7fd09415bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd09415d520 0x7fd0941629e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940b8e60 con 0x7fd09415d110 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0940d18b0 con 0x7fd080004800 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09b7e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0940b8fe0 con 0x7fd09415d520 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40048/0 (socket says 172.21.15.150:40048) 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 -- 172.21.15.150:0/4095804505 learned_addr learned my addr 172.21.15.150:0/4095804505 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:19.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd099d80640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd09415d520 0x7fd0941629e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd09957f640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd080004800 0x7fd09415bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 -- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd09415d520 msgr2=0x7fd0941629e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd09415d520 0x7fd0941629e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 -- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd080004800 msgr2=0x7fd09415bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd080004800 0x7fd09415bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.474+0000 7fd098d7e640 1 -- 172.21.15.150:0/4095804505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09415dbe0 con 0x7fd09415d110 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd098d7e640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd084034040 tx=0x7fd084004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084040020 con 0x7fd09415d110 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd084003b60 con 0x7fd09415d110 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09415dec0 con 0x7fd09415d110 2024-04-25T14:03:19.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084036040 con 0x7fd09415d110 2024-04-25T14:03:19.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.475+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0940aff90 con 0x7fd09415d110 2024-04-25T14:03:19.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.476+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09415e3e0 con 0x7fd09415d110 2024-04-25T14:03:19.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.477+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd084043020 con 0x7fd09415d110 2024-04-25T14:03:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.477+0000 7fd089ffb640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 0x7fd0780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.477+0000 7fd09957f640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 0x7fd0780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.477+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fd0840b55f0 con 0x7fd09415d110 2024-04-25T14:03:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.477+0000 7fd09957f640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 0x7fd0780769c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd090009670 tx=0x7fd09000b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.482+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd09415e3e0 con 0x7fd09415d110 2024-04-25T14:03:19.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.721+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fd09415d970 con 0x7fd09415d110 2024-04-25T14:03:19.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.724+0000 7fd089ffb640 1 -- 172.21.15.150:0/4095804505 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fd08404a030 con 0x7fd09415d110 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 msgr2=0x7fd0780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 0x7fd0780769c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd090009670 tx=0x7fd09000b040 comp rx=0 tx=0).stop 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 msgr2=0x7fd0941624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd084034040 tx=0x7fd084004730 comp rx=0 tx=0).stop 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 shutdown_connections 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0780744b0 0x7fd0780769c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd09415d520 0x7fd0941629e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd09415d110 0x7fd0941624a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 --2- 172.21.15.150:0/4095804505 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd080004800 0x7fd09415bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 >> 172.21.15.150:0/4095804505 conn(0x7fd0940d3960 msgr2=0x7fd0940d0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:19.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 shutdown_connections 2024-04-25T14:03:19.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.726+0000 7fd09b7e3640 1 -- 172.21.15.150:0/4095804505 wait complete. 2024-04-25T14:03:19.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-25T14:03:19.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.850+0000 7f9c85b57640 1 Processor -- start 2024-04-25T14:03:19.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.850+0000 7f9c85b57640 1 -- start start 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c800b2f20 0x7f9c800733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9c80161bf0 0x7f9c80162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9c8015ee20 con 0x7f9c800b2f20 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9c8015e7f0 con 0x7f9c80161bf0 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.851+0000 7f9c85b57640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c80162920 con 0x7f9c8015f2b0 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c800b2f20 0x7f9c800733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9c800b2f20 0x7f9c800733e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40050/0 (socket says 172.21.15.150:40050) 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 -- 172.21.15.150:0/2937760183 learned_addr learned my addr 172.21.15.150:0/2937760183 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7ffff640 1 --2- 172.21.15.150:0/2937760183 >> v2:172.21.15.187:3300/0 conn(0x7f9c80161bf0 0x7f9c80162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7f7fe640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 -- 172.21.15.150:0/2937760183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 msgr2=0x7f9c80072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80072ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 -- 172.21.15.150:0/2937760183 >> v2:172.21.15.187:3300/0 conn(0x7f9c80161bf0 msgr2=0x7f9c80162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 --2- 172.21.15.150:0/2937760183 >> v2:172.21.15.187:3300/0 conn(0x7f9c80161bf0 0x7f9c80162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.852+0000 7f9c7effd640 1 -- 172.21.15.150:0/2937760183 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8015eab0 con 0x7f9c800b2f20 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c7effd640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800733e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9c80071300 tx=0x7f9c7402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473c02cd4cc905ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c7e7fc640 1 -- 172.21.15.150:0/2937760183 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74037070 con 0x7f9c800b2f20 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c7e7fc640 1 -- 172.21.15.150:0/2937760183 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c74032440 con 0x7f9c800b2f20 2024-04-25T14:03:19.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c7e7fc640 1 -- 172.21.15.150:0/2937760183 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c74032810 con 0x7f9c800b2f20 2024-04-25T14:03:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c85b57640 1 -- 172.21.15.150:0/2937760183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 msgr2=0x7f9c800733e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.853+0000 7f9c85b57640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800733e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9c80071300 tx=0x7f9c7402a140 comp rx=0 tx=0).stop 2024-04-25T14:03:19.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 -- 172.21.15.150:0/2937760183 shutdown_connections 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 --2- 172.21.15.150:0/2937760183 >> v2:172.21.15.187:3300/0 conn(0x7f9c80161bf0 0x7f9c80162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800733e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 --2- 172.21.15.150:0/2937760183 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80072ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 -- 172.21.15.150:0/2937760183 >> 172.21.15.150:0/2937760183 conn(0x7f9c800d39b0 msgr2=0x7f9c800d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 -- 172.21.15.150:0/2937760183 shutdown_connections 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.854+0000 7f9c85b57640 1 -- 172.21.15.150:0/2937760183 wait complete. 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.855+0000 7f9c85b57640 1 Processor -- start 2024-04-25T14:03:19.856 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.855+0000 7f9c85b57640 1 -- start start 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c80161bf0 0x7f9c80159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9c800702a0 con 0x7f9c800b2f20 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9c80070120 con 0x7f9c80161bf0 2024-04-25T14:03:19.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c85b57640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9c80070420 con 0x7f9c8015f2b0 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40066/0 (socket says 172.21.15.150:40066) 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 -- 172.21.15.150:0/3259385442 learned_addr learned my addr 172.21.15.150:0/3259385442 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7effd640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7ffff640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c80161bf0 0x7f9c80159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 -- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 msgr2=0x7f9c80159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 -- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c80161bf0 msgr2=0x7f9c80159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c80161bf0 0x7f9c80159860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 -- 172.21.15.150:0/3259385442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c801f7c10 con 0x7f9c800b2f20 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.857+0000 7f9c7f7fe640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000e840 tx=0x7f9c7000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.858+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c70017070 con 0x7f9c800b2f20 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.858+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c7000c730 con 0x7f9c800b2f20 2024-04-25T14:03:19.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.858+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7000cb00 con 0x7f9c800b2f20 2024-04-25T14:03:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.858+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c801f7ef0 con 0x7f9c800b2f20 2024-04-25T14:03:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.858+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c801f84b0 con 0x7f9c800b2f20 2024-04-25T14:03:19.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.859+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c801f8890 con 0x7f9c800b2f20 2024-04-25T14:03:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.859+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9c7001a070 con 0x7f9c800b2f20 2024-04-25T14:03:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.860+0000 7f9c67fff640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 0x7f9c6c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.860+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f9c70094a70 con 0x7f9c800b2f20 2024-04-25T14:03:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.860+0000 7f9c7effd640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 0x7f9c6c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:19.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.860+0000 7f9c7effd640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 0x7f9c6c0769c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c74034000 tx=0x7f9c7402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:19.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:19.864+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9c801f8890 con 0x7f9c800b2f20 2024-04-25T14:03:20.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.107+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f9c80076a40 con 0x7f9c800b2f20 2024-04-25T14:03:20.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.112+0000 7f9c67fff640 1 -- 172.21.15.150:0/3259385442 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f9c7005ed70 con 0x7f9c800b2f20 2024-04-25T14:03:20.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 msgr2=0x7f9c6c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 0x7f9c6c0769c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f9c74034000 tx=0x7f9c7402e000 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 msgr2=0x7f9c800c7c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9c7000e840 tx=0x7f9c7000ed00 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 shutdown_connections 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9c6c0744b0 0x7f9c6c0769c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9c80161bf0 0x7f9c80159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9c8015f2b0 0x7f9c80159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 --2- 172.21.15.150:0/3259385442 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9c800b2f20 0x7f9c800c7c50 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.113+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 >> 172.21.15.150:0/3259385442 conn(0x7f9c800d39b0 msgr2=0x7f9c800b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.114+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 shutdown_connections 2024-04-25T14:03:20.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.114+0000 7f9c85b57640 1 -- 172.21.15.150:0/3259385442 wait complete. 2024-04-25T14:03:20.125 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:20.232 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.232+0000 7fbf8c1c5640 1 Processor -- start 2024-04-25T14:03:20.233 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.232+0000 7fbf8c1c5640 1 -- start start 2024-04-25T14:03:20.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8c1c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf840b4940 0x7fbf840d25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8c1c5640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbf8406e4d0 0x7fbf84167da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8c1c5640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf840d2ee0 con 0x7fbf840b3aa0 2024-04-25T14:03:20.236 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8c1c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbf840d1030 con 0x7fbf840b4940 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8c1c5640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbf84161b60 con 0x7fbf8406e4d0 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf840b4940 0x7fbf840d25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbf840b4940 0x7fbf840d25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40068/0 (socket says 172.21.15.150:40068) 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 -- 172.21.15.150:0/3442443371 learned_addr learned my addr 172.21.15.150:0/3442443371 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf8a762640 1 --2- 172.21.15.150:0/3442443371 >> v2:172.21.15.187:3300/0 conn(0x7fbf8406e4d0 0x7fbf84167da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 -- 172.21.15.150:0/3442443371 >> v2:172.21.15.187:3300/0 conn(0x7fbf8406e4d0 msgr2=0x7fbf84167da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 --2- 172.21.15.150:0/3442443371 >> v2:172.21.15.187:3300/0 conn(0x7fbf8406e4d0 0x7fbf84167da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 -- 172.21.15.150:0/3442443371 >> v1:172.21.15.187:6789/0 conn(0x7fbf840b3aa0 legacy=0x7fbf840b3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.234+0000 7fbf89f61640 1 -- 172.21.15.150:0/3442443371 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf840d31c0 con 0x7fbf840b4940 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.235+0000 7fbf89f61640 1 --2- 172.21.15.150:0/3442443371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b4940 0x7fbf840d25d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbf840b57d0 tx=0x7fbf8002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35b5b6d98343b51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.235+0000 7fbf88f5f640 1 -- 172.21.15.150:0/3442443371 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80036070 con 0x7fbf840b4940 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.235+0000 7fbf88f5f640 1 -- 172.21.15.150:0/3442443371 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80031600 con 0x7fbf840b4940 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.235+0000 7fbf88f5f640 1 -- 172.21.15.150:0/3442443371 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf800319d0 con 0x7fbf840b4940 2024-04-25T14:03:20.237 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/3442443371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b4940 msgr2=0x7fbf840d25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/3442443371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b4940 0x7fbf840d25d0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbf840b57d0 tx=0x7fbf8002a6a0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/3442443371 shutdown_connections 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/3442443371 >> v2:172.21.15.187:3300/0 conn(0x7fbf8406e4d0 0x7fbf84167da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/3442443371 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b4940 0x7fbf840d25d0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/3442443371 >> 172.21.15.150:0/3442443371 conn(0x7fbf8409b4a0 msgr2=0x7fbf8409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/3442443371 shutdown_connections 2024-04-25T14:03:20.238 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.236+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/3442443371 wait complete. 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.238+0000 7fbf8c1c5640 1 Processor -- start 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.238+0000 7fbf8c1c5640 1 -- start start 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf8406e4d0 0x7fbf8414ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf840b4940 0x7fbf840b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbf840b45c0 con 0x7fbf840b3aa0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8406cfa0 con 0x7fbf8406e4d0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8c1c5640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8414f140 con 0x7fbf840b4940 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40076/0 (socket says 172.21.15.150:40076) 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 -- 172.21.15.150:0/2610900612 learned_addr learned my addr 172.21.15.150:0/2610900612 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf8a762640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf840b4940 0x7fbf840b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 -- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf840b4940 msgr2=0x7fbf840b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89f61640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf8406e4d0 0x7fbf8414ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf840b4940 0x7fbf840b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 -- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf8406e4d0 msgr2=0x7fbf8414ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf8406e4d0 0x7fbf8414ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 -- 172.21.15.150:0/2610900612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf840af780 con 0x7fbf840b3aa0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.239+0000 7fbf89760640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400c3f0 tx=0x7fbf7400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.240+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf740052d0 con 0x7fbf840b3aa0 2024-04-25T14:03:20.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.240+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf74005490 con 0x7fbf840b3aa0 2024-04-25T14:03:20.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.240+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf840afa20 con 0x7fbf840b3aa0 2024-04-25T14:03:20.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.240+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74007260 con 0x7fbf840b3aa0 2024-04-25T14:03:20.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.240+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf840affe0 con 0x7fbf840b3aa0 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.241+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf84076a40 con 0x7fbf840b3aa0 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.241+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbf74007400 con 0x7fbf840b3aa0 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.242+0000 7fbf7a7fc640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 0x7fbf680769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.242+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fbf74094520 con 0x7fbf840b3aa0 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.242+0000 7fbf89f61640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 0x7fbf680769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.242+0000 7fbf89f61640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 0x7fbf680769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbf80033000 tx=0x7fbf80009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.246+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf840af780 con 0x7fbf840b3aa0 2024-04-25T14:03:20.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.419+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf8406f160 con 0x7fbf840b3aa0 2024-04-25T14:03:20.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.420+0000 7fbf7a7fc640 1 -- 172.21.15.150:0/2610900612 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) ==== 74+0+13164 (secure 0 0 0) 0x7fbf7405ee70 con 0x7fbf840b3aa0 2024-04-25T14:03:20.420 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:20.420 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":34,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:17.434193+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:20.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 msgr2=0x7fbf680769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 0x7fbf680769c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbf80033000 tx=0x7fbf80009000 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 msgr2=0x7fbf8409fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400c3f0 tx=0x7fbf7400c8b0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 shutdown_connections 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbf680744b0 0x7fbf680769c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbf840b4940 0x7fbf840b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbf840b3aa0 0x7fbf8409fea0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 --2- 172.21.15.150:0/2610900612 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbf8406e4d0 0x7fbf8414ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 >> 172.21.15.150:0/2610900612 conn(0x7fbf8409b4a0 msgr2=0x7fbf8406f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 shutdown_connections 2024-04-25T14:03:20.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.422+0000 7fbf8c1c5640 1 -- 172.21.15.150:0/2610900612 wait complete. 2024-04-25T14:03:20.433 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:03:20.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:03:20.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.545+0000 7f76f3eaa640 1 Processor -- start 2024-04-25T14:03:20.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.545+0000 7f76f3eaa640 1 -- start start 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f3eaa640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76ec070830 0x7f76ec0d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f3eaa640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f76ec15e360 0x7f76ec0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f3eaa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76ec0d15d0 con 0x7f76ec0cf3d0 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f3eaa640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f76ec0ce440 con 0x7f76ec070830 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f3eaa640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f76ec0d1870 con 0x7f76ec15e360 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f1445640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76ec070830 0x7f76ec0d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f1445640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f76ec070830 0x7f76ec0d2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40084/0 (socket says 172.21.15.150:40084) 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f1c46640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f76ec0cf3d0 0x7f76ec070120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39082/0 (socket says 172.21.15.150:39082) 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f1445640 1 -- 172.21.15.150:0/3293730096 learned_addr learned my addr 172.21.15.150:0/3293730096 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:20.547 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.546+0000 7f76f2447640 1 --2- 172.21.15.150:0/3293730096 >> v2:172.21.15.187:3300/0 conn(0x7f76ec15e360 0x7f76ec0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f1445640 1 -- 172.21.15.150:0/3293730096 >> v2:172.21.15.187:3300/0 conn(0x7f76ec15e360 msgr2=0x7f76ec0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:20.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f1445640 1 --2- 172.21.15.150:0/3293730096 >> v2:172.21.15.187:3300/0 conn(0x7f76ec15e360 0x7f76ec0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f1445640 1 -- 172.21.15.150:0/3293730096 >> v1:172.21.15.150:6789/0 conn(0x7f76ec0cf3d0 legacy=0x7f76ec070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f1445640 1 -- 172.21.15.150:0/3293730096 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec0d1b50 con 0x7f76ec070830 2024-04-25T14:03:20.548 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f0c44640 1 -- 172.21.15.150:0/3293730096 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3033957554 0 0) 0x7f76ec0d15d0 con 0x7f76ec0cf3d0 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.547+0000 7f76f1445640 1 --2- 172.21.15.150:0/3293730096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec070830 0x7f76ec0d2070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f76ec162e20 tx=0x7f76dc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee335d9323f527d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f0c44640 1 -- 172.21.15.150:0/3293730096 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc037070 con 0x7f76ec070830 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f0c44640 1 -- 172.21.15.150:0/3293730096 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76dc032440 con 0x7f76ec070830 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f0c44640 1 -- 172.21.15.150:0/3293730096 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76dc032810 con 0x7f76ec070830 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 -- 172.21.15.150:0/3293730096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec070830 msgr2=0x7f76ec0d2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/3293730096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec070830 0x7f76ec0d2070 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f76ec162e20 tx=0x7f76dc02a270 comp rx=0 tx=0).stop 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 -- 172.21.15.150:0/3293730096 shutdown_connections 2024-04-25T14:03:20.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/3293730096 >> v2:172.21.15.187:3300/0 conn(0x7f76ec15e360 0x7f76ec0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/3293730096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec070830 0x7f76ec0d2070 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 -- 172.21.15.150:0/3293730096 >> 172.21.15.150:0/3293730096 conn(0x7f76ec0d3960 msgr2=0x7f76ec0d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 -- 172.21.15.150:0/3293730096 shutdown_connections 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.548+0000 7f76f3eaa640 1 -- 172.21.15.150:0/3293730096 wait complete. 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.550+0000 7f76f3eaa640 1 Processor -- start 2024-04-25T14:03:20.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.550+0000 7f76f3eaa640 1 -- start start 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76ec0cf3d0 0x7f76ec14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76ec15e360 0x7f76ec14cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec070eb0 con 0x7f76ec15fda0 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f76ec070d30 con 0x7f76ec0cf3d0 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.551+0000 7f76f3eaa640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f76ec0cdfb0 con 0x7f76ec15e360 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40086/0 (socket says 172.21.15.150:40086) 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 -- 172.21.15.150:0/2787150859 learned_addr learned my addr 172.21.15.150:0/2787150859 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f1c46640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76ec0cf3d0 0x7f76ec14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f1445640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76ec15e360 0x7f76ec14cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 -- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76ec15e360 msgr2=0x7f76ec14cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76ec15e360 0x7f76ec14cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 -- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76ec0cf3d0 msgr2=0x7f76ec14a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76ec0cf3d0 0x7f76ec14a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 -- 172.21.15.150:0/2787150859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76ec1606f0 con 0x7f76ec15fda0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f2447640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009cb0 tx=0x7f76e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4018020 con 0x7f76ec15fda0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76e4014480 con 0x7f76ec15fda0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76ec0cd020 con 0x7f76ec15fda0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4014870 con 0x7f76ec15fda0 2024-04-25T14:03:20.554 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.552+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76ec14de20 con 0x7f76ec15fda0 2024-04-25T14:03:20.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.553+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76ec14d890 con 0x7f76ec15fda0 2024-04-25T14:03:20.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.554+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f76e4005d10 con 0x7f76ec15fda0 2024-04-25T14:03:20.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.554+0000 7f76e27fc640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 0x7f76d00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.555+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f76e4095da0 con 0x7f76ec15fda0 2024-04-25T14:03:20.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.555+0000 7f76f1c46640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 0x7f76d00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.555+0000 7f76f1c46640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 0x7f76d00769c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000f60 tx=0x7f76e8007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.559+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f76e4022030 con 0x7f76ec15fda0 2024-04-25T14:03:20.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.738+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f76ec0b24c0 con 0x7f76ec15fda0 2024-04-25T14:03:20.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.738+0000 7f76e27fc640 1 -- 172.21.15.150:0/2787150859 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) ==== 93+0+10 (secure 0 0 0) 0x7f76ec1606f0 con 0x7f76ec15fda0 2024-04-25T14:03:20.739 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.740+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 msgr2=0x7f76d00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 0x7f76d00769c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f76e8000f60 tx=0x7f76e8007c00 comp rx=0 tx=0).stop 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 msgr2=0x7f76ec1601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f76e4009cb0 tx=0x7f76e400c490 comp rx=0 tx=0).stop 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 shutdown_connections 2024-04-25T14:03:20.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f76d00744b0 0x7f76d00769c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f76ec15fda0 0x7f76ec1601b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f76ec15e360 0x7f76ec14cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 --2- 172.21.15.150:0/2787150859 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f76ec0cf3d0 0x7f76ec14a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 >> 172.21.15.150:0/2787150859 conn(0x7f76ec0d3960 msgr2=0x7f76ec0cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 shutdown_connections 2024-04-25T14:03:20.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.741+0000 7f76f3eaa640 1 -- 172.21.15.150:0/2787150859 wait complete. 2024-04-25T14:03:20.752 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T14:03:20.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.858+0000 7fe8a009f640 1 Processor -- start 2024-04-25T14:03:20.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.858+0000 7fe8a009f640 1 -- start start 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe8a009f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe8980d1120 0x7fe898070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe8a009f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980aec30 con 0x7fe898161b10 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe8a009f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8980aef10 con 0x7fe898154950 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe8a009f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe8980d1530 con 0x7fe8980d1120 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89d63a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe898161b10 0x7fe898161f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39094/0 (socket says 172.21.15.150:39094) 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89d63a640 1 -- 172.21.15.150:0/2511841588 learned_addr learned my addr 172.21.15.150:0/2511841588 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3319432776 0 0) 0x7fe8980aec30 con 0x7fe898161b10 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2511841588 >> v2:172.21.15.187:3300/0 conn(0x7fe8980d1120 0x7fe898070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe884003520 con 0x7fe898161b10 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4077856702 0 0) 0x7fe884003520 con 0x7fe898161b10 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 >> v2:172.21.15.187:3300/0 conn(0x7fe8980d1120 msgr2=0x7fe898070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:20.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 --2- 172.21.15.150:0/2511841588 >> v2:172.21.15.187:3300/0 conn(0x7fe8980d1120 0x7fe898070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.860+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 >> v1:172.21.15.187:6789/0 conn(0x7fe898154950 legacy=0x7fe898154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8980af1f0 con 0x7fe898161b10 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fe888001670 con 0x7fe898161b10 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 >> v1:172.21.15.150:6789/0 conn(0x7fe898161b10 legacy=0x7fe898161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8840039b0 0x7fe884003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 0x7fe884004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe884005350 0x7fe88400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980d1530 con 0x7fe884004800 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe88400c280 con 0x7fe8840039b0 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe88400c480 con 0x7fe884005350 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 0x7fe884004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8840039b0 0x7fe884003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89e63c640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe884005350 0x7fe88400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 -- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe884005350 msgr2=0x7fe88400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe884005350 0x7fe88400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 -- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8840039b0 msgr2=0x7fe884003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8840039b0 0x7fe884003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89d63a640 1 -- 172.21.15.150:0/2511841588 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe884003520 con 0x7fe884004800 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.861+0000 7fe89e63c640 1 -- 172.21.15.150:0/2511841588 reap_dead start 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.862+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 0x7fe884004c10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c240 tx=0x7fe884005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ef00733052a8fe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.862+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888007d10 con 0x7fe884004800 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.862+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8880058d0 con 0x7fe884004800 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.862+0000 7fe89ce39640 1 -- 172.21.15.150:0/2511841588 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888005d40 con 0x7fe884004800 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 -- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 msgr2=0x7fe884004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 0x7fe884004c10 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fe88800c240 tx=0x7fe884005830 comp rx=0 tx=0).stop 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 -- 172.21.15.150:0/2511841588 shutdown_connections 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2511841588 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe884004800 0x7fe884004c10 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 -- 172.21.15.150:0/2511841588 >> 172.21.15.150:0/2511841588 conn(0x7fe8980d1ae0 msgr2=0x7fe8980d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:20.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 -- 172.21.15.150:0/2511841588 shutdown_connections 2024-04-25T14:03:20.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.863+0000 7fe8a009f640 1 -- 172.21.15.150:0/2511841588 wait complete. 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.864+0000 7fe8a009f640 1 Processor -- start 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.864+0000 7fe8a009f640 1 -- start start 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 0x7fe898153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe89815e390 0x7fe89815e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980ae960 con 0x7fe89815b430 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe8980ae7e0 con 0x7fe884004800 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe898161f20 con 0x7fe89815e390 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40110/0 (socket says 172.21.15.150:40110) 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 -- 172.21.15.150:0/2613010184 learned_addr learned my addr 172.21.15.150:0/2613010184 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 0x7fe898153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89e63c640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe89815e390 0x7fe89815e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 -- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe89815e390 msgr2=0x7fe89815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe89815e390 0x7fe89815e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 -- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 msgr2=0x7fe898153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 0x7fe898153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 -- 172.21.15.150:0/2613010184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8981e9c70 con 0x7fe89815b430 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 0x7fe898153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe89d63a640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe888001970 tx=0x7fe8980af7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888040070 con 0x7fe89815b430 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe888000ed0 con 0x7fe89815b430 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe89815ece0 con 0x7fe89815b430 2024-04-25T14:03:20.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.866+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe888007d10 con 0x7fe89815b430 2024-04-25T14:03:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.867+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8981e8eb0 con 0x7fe89815b430 2024-04-25T14:03:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.868+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe89815bd80 con 0x7fe89815b430 2024-04-25T14:03:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.868+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe888043070 con 0x7fe89815b430 2024-04-25T14:03:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.868+0000 7fe88e7fc640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 0x7fe87c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:20.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.869+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7fe8880b5020 con 0x7fe89815b430 2024-04-25T14:03:20.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.869+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 0x7fe87c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:20.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.869+0000 7fe89de3b640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 0x7fe87c0769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe894009bc0 tx=0x7fe894008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:20.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:20.873+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe89815bd80 con 0x7fe89815b430 2024-04-25T14:03:21.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.048+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fe898076a40 con 0x7fe89815b430 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.049+0000 7fe88e7fc640 1 -- 172.21.15.150:0/2613010184 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v34) ==== 84+0+65 (secure 0 0 0) 0x7fe88804a030 con 0x7fe89815b430 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:03:21.049 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-04-25T14:03:21.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 msgr2=0x7fe87c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 0x7fe87c0769f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe894009bc0 tx=0x7fe894008040 comp rx=0 tx=0).stop 2024-04-25T14:03:21.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 msgr2=0x7fe89815b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fe888001970 tx=0x7fe8980af7b0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 shutdown_connections 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe87c0744e0 0x7fe87c0769f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe89815e390 0x7fe89815e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe89815b430 0x7fe89815b840 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 --2- 172.21.15.150:0/2613010184 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe884004800 0x7fe898153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 >> 172.21.15.150:0/2613010184 conn(0x7fe8980d1ae0 msgr2=0x7fe89815d1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 shutdown_connections 2024-04-25T14:03:21.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.051+0000 7fe8a009f640 1 -- 172.21.15.150:0/2613010184 wait complete. 2024-04-25T14:03:21.062 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T14:03:21.062 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T14:03:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.174+0000 7f8b85ea7640 1 Processor -- start 2024-04-25T14:03:21.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.174+0000 7f8b85ea7640 1 -- start start 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b85ea7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 0x7f8b800b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b85ea7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b80161be0 0x7f8b80161ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b85ea7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b800d3240 con 0x7f8b800d2e30 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b85ea7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8b800b3260 con 0x7f8b80161be0 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b85ea7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b800d29a0 con 0x7f8b800b5b30 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8b80161be0 0x7f8b80161ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7f7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f8b800d2e30 0x7f8b800b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39104/0 (socket says 172.21.15.150:39104) 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7f7fe640 1 -- 172.21.15.150:0/619286872 learned_addr learned my addr 172.21.15.150:0/619286872 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7effd640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 0x7f8b800b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7ffff640 1 -- 172.21.15.150:0/619286872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 msgr2=0x7f8b800b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7ffff640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 0x7f8b800b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.175+0000 7f8b7ffff640 1 -- 172.21.15.150:0/619286872 >> v1:172.21.15.150:6789/0 conn(0x7f8b800d2e30 legacy=0x7f8b800b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7ffff640 1 -- 172.21.15.150:0/619286872 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b80162880 con 0x7f8b80161be0 2024-04-25T14:03:21.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7effd640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 0x7f8b800b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7ffff640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b80161ff0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097d0 tx=0x7f8b7802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb9b5caa9b9038 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7e7fc640 1 -- 172.21.15.150:0/619286872 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78037070 con 0x7f8b80161be0 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7e7fc640 1 -- 172.21.15.150:0/619286872 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b78032440 con 0x7f8b80161be0 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.176+0000 7f8b7e7fc640 1 -- 172.21.15.150:0/619286872 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78032810 con 0x7f8b80161be0 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 -- 172.21.15.150:0/619286872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 msgr2=0x7f8b80161ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b80161ff0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f8b780097d0 tx=0x7f8b7802a250 comp rx=0 tx=0).stop 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 -- 172.21.15.150:0/619286872 shutdown_connections 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b800b5b30 0x7f8b800b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/619286872 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b80161ff0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 -- 172.21.15.150:0/619286872 >> 172.21.15.150:0/619286872 conn(0x7f8b800d0420 msgr2=0x7f8b800d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:21.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 -- 172.21.15.150:0/619286872 shutdown_connections 2024-04-25T14:03:21.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.177+0000 7f8b85ea7640 1 -- 172.21.15.150:0/619286872 wait complete. 2024-04-25T14:03:21.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.179+0000 7f8b85ea7640 1 Processor -- start 2024-04-25T14:03:21.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.179+0000 7f8b85ea7640 1 -- start start 2024-04-25T14:03:21.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 0x7f8b8014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b8014d890 0x7f8b8014b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8b80162530 con 0x7f8b80161be0 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8b800d2630 con 0x7f8b800d2e30 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b85ea7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8b800b6d70 con 0x7f8b8014d890 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b7effd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40118/0 (socket says 172.21.15.150:40118) 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.180+0000 7f8b7effd640 1 -- 172.21.15.150:0/294324998 learned_addr learned my addr 172.21.15.150:0/294324998 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7ffff640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b8014d890 0x7f8b8014b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7f7fe640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 0x7f8b8014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 -- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b8014d890 msgr2=0x7f8b8014b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b8014d890 0x7f8b8014b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 -- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 msgr2=0x7f8b8014ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 0x7f8b8014ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 -- 172.21.15.150:0/294324998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8014dca0 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7f7fe640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 0x7f8b8014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b7effd640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400d7d0 tx=0x7f8b7400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b74004030 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b740041d0 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b80160720 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7400b6c0 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.181+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b80160ce0 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.182+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8015e490 con 0x7f8b80161be0 2024-04-25T14:03:21.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.183+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8b7400b860 con 0x7f8b80161be0 2024-04-25T14:03:21.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.183+0000 7f8b67fff640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 0x7f8b6c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.183+0000 7f8b7f7fe640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 0x7f8b6c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.183+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 5249+0+0 (secure 0 0 0) 0x7f8b740952b0 con 0x7f8b80161be0 2024-04-25T14:03:21.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.184+0000 7f8b7f7fe640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 0x7f8b6c0769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b700012e0 tx=0x7f8b70009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.188+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b8015e490 con 0x7f8b80161be0 2024-04-25T14:03:21.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.360+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8b801604b0 con 0x7f8b80161be0 2024-04-25T14:03:21.439 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.439+0000 7f8b67fff640 1 -- 172.21.15.150:0/294324998 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v35) ==== 138+0+0 (secure 0 0 0) 0x7f8b7405fb80 con 0x7f8b80161be0 2024-04-25T14:03:21.439 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-04-25T14:03:21.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 msgr2=0x7f8b6c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 0x7f8b6c0769c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8b700012e0 tx=0x7f8b70009040 comp rx=0 tx=0).stop 2024-04-25T14:03:21.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 msgr2=0x7f8b8014d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400d7d0 tx=0x7f8b7400dc90 comp rx=0 tx=0).stop 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 shutdown_connections 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8b6c0744b0 0x7f8b6c0769c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8b8014d890 0x7f8b8014b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8b80161be0 0x7f8b8014d350 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 --2- 172.21.15.150:0/294324998 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8b800d2e30 0x7f8b8014ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 >> 172.21.15.150:0/294324998 conn(0x7f8b800d0420 msgr2=0x7f8b800d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 shutdown_connections 2024-04-25T14:03:21.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.441+0000 7f8b85ea7640 1 -- 172.21.15.150:0/294324998 wait complete. 2024-04-25T14:03:21.456 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T14:03:21.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.570+0000 7fa6b09ee640 1 Processor -- start 2024-04-25T14:03:21.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.570+0000 7fa6b09ee640 1 -- start start 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa6ac06c570 0x7fa6ac06daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6ac160330 0x7fa6ac0d3f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac0d48b0 0x7fa6ac0d4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa6ac160740 con 0x7fa6ac160330 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa6ac0d5200 con 0x7fa6ac06c570 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6b09ee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa6ac06ca50 con 0x7fa6ac0d48b0 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6ac160330 0x7fa6ac0d3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa6ac160330 0x7fa6ac0d3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40130/0 (socket says 172.21.15.150:40130) 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 -- 172.21.15.150:0/2840751789 learned_addr learned my addr 172.21.15.150:0/2840751789 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6aa59c640 1 --2- 172.21.15.150:0/2840751789 >> v2:172.21.15.187:3300/0 conn(0x7fa6ac06c570 0x7fa6ac06daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6aad9d640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac0d48b0 0x7fa6ac0d4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 -- 172.21.15.150:0/2840751789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac0d48b0 msgr2=0x7fa6ac0d4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac0d48b0 0x7fa6ac0d4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 -- 172.21.15.150:0/2840751789 >> v2:172.21.15.187:3300/0 conn(0x7fa6ac06c570 msgr2=0x7fa6ac06daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 --2- 172.21.15.150:0/2840751789 >> v2:172.21.15.187:3300/0 conn(0x7fa6ac06c570 0x7fa6ac06daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.572+0000 7fa6a9d9b640 1 -- 172.21.15.150:0/2840751789 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac06cd30 con 0x7fa6ac160330 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.573+0000 7fa6a9d9b640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac0d3f90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac08eb10 tx=0x7fa6a002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6b016782aaf98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.573+0000 7fa6a959a640 1 -- 172.21.15.150:0/2840751789 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0037070 con 0x7fa6ac160330 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.573+0000 7fa6a959a640 1 -- 172.21.15.150:0/2840751789 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a0032440 con 0x7fa6ac160330 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.573+0000 7fa6a959a640 1 -- 172.21.15.150:0/2840751789 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0032810 con 0x7fa6ac160330 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 -- 172.21.15.150:0/2840751789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 msgr2=0x7fa6ac0d3f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac0d3f90 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac08eb10 tx=0x7fa6a002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 -- 172.21.15.150:0/2840751789 shutdown_connections 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac0d48b0 0x7fa6ac0d4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/2840751789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac0d3f90 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/2840751789 >> v2:172.21.15.187:3300/0 conn(0x7fa6ac06c570 0x7fa6ac06daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 -- 172.21.15.150:0/2840751789 >> 172.21.15.150:0/2840751789 conn(0x7fa6ac0d63f0 msgr2=0x7fa6ac0d2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 -- 172.21.15.150:0/2840751789 shutdown_connections 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.574+0000 7fa6b09ee640 1 -- 172.21.15.150:0/2840751789 wait complete. 2024-04-25T14:03:21.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.576+0000 7fa6b09ee640 1 Processor -- start 2024-04-25T14:03:21.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.576+0000 7fa6b09ee640 1 -- start start 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac06c570 0x7fa6ac1fa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6ac0d48b0 0x7fa6ac163030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac0d44d0 con 0x7fa6ac160330 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa6ac06d2f0 con 0x7fa6ac0d48b0 2024-04-25T14:03:21.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.577+0000 7fa6b09ee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa6ac0d4650 con 0x7fa6ac06c570 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40134/0 (socket says 172.21.15.150:40134) 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 -- 172.21.15.150:0/3299922340 learned_addr learned my addr 172.21.15.150:0/3299922340 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6a9d9b640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6ac0d48b0 0x7fa6ac163030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aa59c640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac06c570 0x7fa6ac1fa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 -- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac06c570 msgr2=0x7fa6ac1fa5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac06c570 0x7fa6ac1fa5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 -- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6ac0d48b0 msgr2=0x7fa6ac163030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6ac0d48b0 0x7fa6ac163030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 -- 172.21.15.150:0/3299922340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6ac0ccc80 con 0x7fa6ac160330 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6aad9d640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7c0 tx=0x7fa6a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4017070 con 0x7fa6ac160330 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6a400c730 con 0x7fa6ac160330 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400cb20 con 0x7fa6ac160330 2024-04-25T14:03:21.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6ac0ccf60 con 0x7fa6ac160330 2024-04-25T14:03:21.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.578+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6ac0cd520 con 0x7fa6ac160330 2024-04-25T14:03:21.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.579+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6ac076a40 con 0x7fa6ac160330 2024-04-25T14:03:21.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.580+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa6a401a070 con 0x7fa6ac160330 2024-04-25T14:03:21.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.580+0000 7fa69affd640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 0x7fa68c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.581+0000 7fa6aa59c640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 0x7fa68c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.581+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5619+0+0 (secure 0 0 0) 0x7fa6a40956c0 con 0x7fa6ac160330 2024-04-25T14:03:21.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.581+0000 7fa6aa59c640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 0x7fa68c0769f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa69c011200 tx=0x7fa69c00d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:21.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.585+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa6a405f930 con 0x7fa6ac160330 2024-04-25T14:03:21.762 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:21.762+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa6ac0baca0 con 0x7fa6ac160330 2024-04-25T14:03:22.441 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.441+0000 7fa69affd640 1 -- 172.21.15.150:0/3299922340 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v36) ==== 170+0+0 (secure 0 0 0) 0x7fa6a4063c30 con 0x7fa6ac160330 2024-04-25T14:03:22.441 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-04-25T14:03:22.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.443+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 msgr2=0x7fa68c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:22.443 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.443+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 0x7fa68c0769f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa69c011200 tx=0x7fa69c00d040 comp rx=0 tx=0).stop 2024-04-25T14:03:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.443+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 msgr2=0x7fa6ac158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.443+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e7c0 tx=0x7fa6a400ec80 comp rx=0 tx=0).stop 2024-04-25T14:03:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 shutdown_connections 2024-04-25T14:03:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa68c0744e0 0x7fa68c0769f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa6ac160330 0x7fa6ac158070 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa6ac0d48b0 0x7fa6ac163030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 --2- 172.21.15.150:0/3299922340 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa6ac06c570 0x7fa6ac1fa5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 >> 172.21.15.150:0/3299922340 conn(0x7fa6ac0d63f0 msgr2=0x7fa6ac0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 shutdown_connections 2024-04-25T14:03:22.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.444+0000 7fa6b09ee640 1 -- 172.21.15.150:0/3299922340 wait complete. 2024-04-25T14:03:22.469 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T14:03:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.581+0000 7f7b66d5f640 1 Processor -- start 2024-04-25T14:03:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.581+0000 7f7b66d5f640 1 -- start start 2024-04-25T14:03:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.582+0000 7f7b66d5f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7b600cf3d0 0x7f7b60070120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.582+0000 7f7b66d5f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7b6015e360 0x7f7b60070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b66d5f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b60161d80 0x7f7b60162190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b66d5f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f7b600d15d0 con 0x7f7b600cf3d0 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b66d5f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7b600ce440 con 0x7f7b6015e360 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b66d5f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7b600d1870 con 0x7f7b60161d80 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7b600cf3d0 0x7f7b60070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f7b600cf3d0 0x7f7b60070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40146/0 (socket says 172.21.15.150:40146) 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 -- 172.21.15.150:0/119594420 learned_addr learned my addr 172.21.15.150:0/119594420 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b652fc640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b60161d80 0x7f7b60162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b5ffff640 1 --2- 172.21.15.150:0/119594420 >> v2:172.21.15.187:3300/0 conn(0x7f7b6015e360 0x7f7b60070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 -- 172.21.15.150:0/119594420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b60161d80 msgr2=0x7f7b60162190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b60161d80 0x7f7b60162190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 -- 172.21.15.150:0/119594420 >> v2:172.21.15.187:3300/0 conn(0x7f7b6015e360 msgr2=0x7f7b60070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 --2- 172.21.15.150:0/119594420 >> v2:172.21.15.187:3300/0 conn(0x7f7b6015e360 0x7f7b60070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.584 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.583+0000 7f7b64afb640 1 -- 172.21.15.150:0/119594420 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b600d1b50 con 0x7f7b600cf3d0 2024-04-25T14:03:22.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.584+0000 7f7b64afb640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b600cf3d0 0x7f7b60070120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7b600754e0 tx=0x7f7b5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9580e67ead38029c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.584+0000 7f7b5f7fe640 1 -- 172.21.15.150:0/119594420 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50037070 con 0x7f7b600cf3d0 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.584+0000 7f7b5f7fe640 1 -- 172.21.15.150:0/119594420 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b50032440 con 0x7f7b600cf3d0 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.584+0000 7f7b5f7fe640 1 -- 172.21.15.150:0/119594420 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b50032810 con 0x7f7b600cf3d0 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 -- 172.21.15.150:0/119594420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b600cf3d0 msgr2=0x7f7b60070120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b600cf3d0 0x7f7b60070120 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f7b600754e0 tx=0x7f7b5002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 -- 172.21.15.150:0/119594420 shutdown_connections 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b60161d80 0x7f7b60162190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/119594420 >> v2:172.21.15.187:3300/0 conn(0x7f7b6015e360 0x7f7b60070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/119594420 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b600cf3d0 0x7f7b60070120 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 -- 172.21.15.150:0/119594420 >> 172.21.15.150:0/119594420 conn(0x7f7b600d3960 msgr2=0x7f7b600d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 -- 172.21.15.150:0/119594420 shutdown_connections 2024-04-25T14:03:22.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.585+0000 7f7b66d5f640 1 -- 172.21.15.150:0/119594420 wait complete. 2024-04-25T14:03:22.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.586+0000 7f7b66d5f640 1 Processor -- start 2024-04-25T14:03:22.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.586+0000 7f7b66d5f640 1 -- start start 2024-04-25T14:03:22.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 0x7f7b6006f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b6015e360 0x7f7b600cd750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60070eb0 con 0x7f7b60161d80 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7b60070d30 con 0x7f7b600cf3d0 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b66d5f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7b600cdfb0 con 0x7f7b6015e360 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40148/0 (socket says 172.21.15.150:40148) 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 -- 172.21.15.150:0/3175702072 learned_addr learned my addr 172.21.15.150:0/3175702072 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b64afb640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 0x7f7b6006f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b5ffff640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b6015e360 0x7f7b600cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 -- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b6015e360 msgr2=0x7f7b600cd750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b6015e360 0x7f7b600cd750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 -- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 msgr2=0x7f7b6006f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 0x7f7b6006f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:22.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b652fc640 1 -- 172.21.15.150:0/3175702072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b6006fd60 con 0x7f7b60161d80 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.588+0000 7f7b64afb640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 0x7f7b6006f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b652fc640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800efc0 tx=0x7f7b5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58018020 con 0x7f7b60161d80 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b58014480 con 0x7f7b60161d80 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58009e60 con 0x7f7b60161d80 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b601f9ca0 con 0x7f7b60161d80 2024-04-25T14:03:22.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.589+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b601fa260 con 0x7f7b60161d80 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.590+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b6006fad0 con 0x7f7b60161d80 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.591+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7b58005d10 con 0x7f7b60161d80 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.591+0000 7f7b5d7fa640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 0x7f7b4c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.591+0000 7f7b64afb640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 0x7f7b4c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.591+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6001+0+0 (secure 0 0 0) 0x7f7b58096430 con 0x7f7b60161d80 2024-04-25T14:03:22.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.592+0000 7f7b64afb640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 0x7f7b4c0769f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7b50034000 tx=0x7f7b5002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:22.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.596+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b58022030 con 0x7f7b60161d80 2024-04-25T14:03:22.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:22.811+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f7b6015dc30 con 0x7f7b60161d80 2024-04-25T14:03:23.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.467+0000 7f7b5d7fa640 1 -- 172.21.15.150:0/3175702072 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7f7b58010040 con 0x7f7b60161d80 2024-04-25T14:03:23.468 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 5 and data pool 6 2024-04-25T14:03:23.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.473+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 msgr2=0x7f7b4c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.473+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 0x7f7b4c0769f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7b50034000 tx=0x7f7b5002e000 comp rx=0 tx=0).stop 2024-04-25T14:03:23.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.473+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 msgr2=0x7f7b600c8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.473+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800efc0 tx=0x7f7b5800c490 comp rx=0 tx=0).stop 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 shutdown_connections 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7b4c0744e0 0x7f7b4c0769f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7b60161d80 0x7f7b600c8de0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7b6015e360 0x7f7b600cd750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 --2- 172.21.15.150:0/3175702072 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7b600cf3d0 0x7f7b6006f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.475+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 >> 172.21.15.150:0/3175702072 conn(0x7f7b600d3960 msgr2=0x7f7b600684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:23.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.476+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 shutdown_connections 2024-04-25T14:03:23.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.476+0000 7f7b66d5f640 1 -- 172.21.15.150:0/3175702072 wait complete. 2024-04-25T14:03:23.495 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:23.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.607+0000 7fdd2a84c640 1 Processor -- start 2024-04-25T14:03:23.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.607+0000 7fdd2a84c640 1 -- start start 2024-04-25T14:03:23.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd2a84c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd24160e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd2a84c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd241613a0 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd2a84c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd240b3f00 con 0x7fdd2415cfe0 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd2a84c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd240b4ad0 con 0x7fdd24160a50 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd23fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdd240b4f60 0x7fdd240cf8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39114/0 (socket says 172.21.15.150:39114) 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.608+0000 7fdd23fff640 1 -- 172.21.15.150:0/2713462153 learned_addr learned my addr 172.21.15.150:0/2713462153 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3554168124 0 0) 0x7fdd14001290 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd237fe640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd24160e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd10003520 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1230267945 0 0) 0x7fdd10003520 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 msgr2=0x7fdd24160e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd24160e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 >> v1:172.21.15.187:6789/0 conn(0x7fdd2415cfe0 legacy=0x7fdd2415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd240b41a0 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fdd140019c0 con 0x7fdd240b4f60 2024-04-25T14:03:23.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 >> v1:172.21.15.150:6789/0 conn(0x7fdd240b4f60 legacy=0x7fdd240cf8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100039e0 0x7fdd10003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd10004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10004d40 0x7fdd10005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd240b4ad0 con 0x7fdd10004d40 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdd1000ba70 con 0x7fdd100039e0 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd1000bc70 con 0x7fdd24160a50 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd28de9640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10004d40 0x7fdd10005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd237fe640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd10004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.609+0000 7fdd23fff640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100039e0 0x7fdd10003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 -- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 msgr2=0x7fdd10004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd24160a50 0x7fdd10004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 -- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100039e0 msgr2=0x7fdd10003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd100039e0 0x7fdd10003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 -- 172.21.15.150:0/2713462153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd10003520 con 0x7fdd10004d40 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd28de9640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10004d40 0x7fdd10005150 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd240b88a0 tx=0x7fdd10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b62b02d2abb450 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.610+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c001b50 con 0x7fdd10004d40 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd1c031600 con 0x7fdd10004d40 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd22ffd640 1 -- 172.21.15.150:0/2713462153 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c0319b0 con 0x7fdd10004d40 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 -- 172.21.15.150:0/2713462153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10004d40 msgr2=0x7fdd10005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/2713462153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd10004d40 0x7fdd10005150 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fdd240b88a0 tx=0x7fdd10005830 comp rx=0 tx=0).stop 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd28de9640 1 -- 172.21.15.150:0/2713462153 reap_dead start 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 -- 172.21.15.150:0/2713462153 shutdown_connections 2024-04-25T14:03:23.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 -- 172.21.15.150:0/2713462153 >> 172.21.15.150:0/2713462153 conn(0x7fdd2409b4a0 msgr2=0x7fdd2409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:23.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 -- 172.21.15.150:0/2713462153 shutdown_connections 2024-04-25T14:03:23.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.611+0000 7fdd2a84c640 1 -- 172.21.15.150:0/2713462153 wait complete. 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.613+0000 7fdd2a84c640 1 Processor -- start 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.613+0000 7fdd2a84c640 1 -- start start 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.614+0000 7fdd2a84c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd241f8670 0x7fdd241f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.614+0000 7fdd2a84c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd240cb940 0x7fdd241f8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdd240d0000 con 0x7fdd240cb530 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdd240b4760 con 0x7fdd241f8670 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdd24161730 con 0x7fdd240cb940 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40170/0 (socket says 172.21.15.150:40170) 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 -- 172.21.15.150:0/150184280 learned_addr learned my addr 172.21.15.150:0/150184280 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:23.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd28de9640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd240cb940 0x7fdd241f8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd23fff640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd241f8670 0x7fdd241f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 -- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd240cb940 msgr2=0x7fdd241f8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd240cb940 0x7fdd241f8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 -- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd241f8670 msgr2=0x7fdd241f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd241f8670 0x7fdd241f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 -- 172.21.15.150:0/150184280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd240ca490 con 0x7fdd240cb530 2024-04-25T14:03:23.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd237fe640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0119a0 tx=0x7fdd0c011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c00faf0 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd0c00fc90 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c014440 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd240ca770 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.615+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd240cad30 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.616+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd24076a40 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.617+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdd0c012040 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.618+0000 7fdd20ff9640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 0x7fdd040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.618+0000 7fdd23fff640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 0x7fdd040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.618+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7fdd0c095940 con 0x7fdd240cb530 2024-04-25T14:03:23.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.618+0000 7fdd23fff640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 0x7fdd040769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdd241f7a30 tx=0x7fdd14001030 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.623+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdd0c019030 con 0x7fdd240cb530 2024-04-25T14:03:23.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.800+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd240cf6d0 con 0x7fdd240cb530 2024-04-25T14:03:23.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.801+0000 7fdd20ff9640 1 -- 172.21.15.150:0/150184280 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7fdd0c05f980 con 0x7fdd240cb530 2024-04-25T14:03:23.801 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:23.802 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:22.813136+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:23.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 msgr2=0x7fdd040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 0x7fdd040769c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdd241f7a30 tx=0x7fdd14001030 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 msgr2=0x7fdd241f7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c0119a0 tx=0x7fdd0c011e60 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 shutdown_connections 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdd040744b0 0x7fdd040769c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdd240cb940 0x7fdd241f8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdd240cb530 0x7fdd241f7ad0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 --2- 172.21.15.150:0/150184280 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdd241f8670 0x7fdd241f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 >> 172.21.15.150:0/150184280 conn(0x7fdd2409b4a0 msgr2=0x7fdd240cd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 shutdown_connections 2024-04-25T14:03:23.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.803+0000 7fdd2a84c640 1 -- 172.21.15.150:0/150184280 wait complete. 2024-04-25T14:03:23.815 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:23.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.922+0000 7f9dd32f0640 1 Processor -- start 2024-04-25T14:03:23.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.922+0000 7f9dd32f0640 1 -- start start 2024-04-25T14:03:23.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.923+0000 7f9dd32f0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9dcc156cd0 0x7f9dcc15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.923+0000 7f9dd32f0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 0x7f9dcc06f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.923+0000 7f9dd32f0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dcc0afe00 con 0x7f9dcc070e70 2024-04-25T14:03:23.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.923+0000 7f9dd32f0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9dcc0b00a0 con 0x7f9dcc156cd0 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.923+0000 7f9dd32f0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9dcc0b0380 con 0x7f9dcc06d510 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd088b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9dcc070e70 0x7f9dcc06ce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39124/0 (socket says 172.21.15.150:39124) 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd088b640 1 -- 172.21.15.150:0/3855857716 learned_addr learned my addr 172.21.15.150:0/3855857716 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd108c640 1 --2- 172.21.15.150:0/3855857716 >> v2:172.21.15.187:3300/0 conn(0x7f9dcc156cd0 0x7f9dcc15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd188d640 1 --2- 172.21.15.150:0/3855857716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 0x7f9dcc06f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 302604925 0 0) 0x7f9dcc0afe00 con 0x7f9dcc070e70 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9db8003520 con 0x7f9dcc070e70 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd188d640 1 -- 172.21.15.150:0/3855857716 >> v2:172.21.15.187:3300/0 conn(0x7f9dcc156cd0 msgr2=0x7f9dcc15d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd188d640 1 --2- 172.21.15.150:0/3855857716 >> v2:172.21.15.187:3300/0 conn(0x7f9dcc156cd0 0x7f9dcc15d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3118130163 0 0) 0x7f9db8003520 con 0x7f9dcc070e70 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd188d640 1 -- 172.21.15.150:0/3855857716 >> v1:172.21.15.150:6789/0 conn(0x7f9dcc070e70 legacy=0x7f9dcc06ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.924+0000 7f9dd188d640 1 -- 172.21.15.150:0/3855857716 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc0b0660 con 0x7f9dcc06d510 2024-04-25T14:03:23.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.925+0000 7f9dd188d640 1 --2- 172.21.15.150:0/3855857716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 0x7f9dcc06f240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400aea0 tx=0x7f9dc402b740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7d0ee7eec72a70a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.925+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4037070 con 0x7f9dcc06d510 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.925+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc4008440 con 0x7f9dcc06d510 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.925+0000 7f9dc3fff640 1 -- 172.21.15.150:0/3855857716 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc001670 con 0x7f9dcc06d510 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 -- 172.21.15.150:0/3855857716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 msgr2=0x7f9dcc06f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/3855857716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 0x7f9dcc06f240 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9dc400aea0 tx=0x7f9dc402b740 comp rx=0 tx=0).stop 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 -- 172.21.15.150:0/3855857716 shutdown_connections 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/3855857716 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc06d510 0x7f9dcc06f240 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/3855857716 >> v2:172.21.15.187:3300/0 conn(0x7f9dcc156cd0 0x7f9dcc15d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 -- 172.21.15.150:0/3855857716 >> 172.21.15.150:0/3855857716 conn(0x7f9dcc0d1ae0 msgr2=0x7f9dcc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 -- 172.21.15.150:0/3855857716 shutdown_connections 2024-04-25T14:03:23.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.926+0000 7f9dd32f0640 1 -- 172.21.15.150:0/3855857716 wait complete. 2024-04-25T14:03:23.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.927+0000 7f9dd32f0640 1 Processor -- start 2024-04-25T14:03:23.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.927+0000 7f9dd32f0640 1 -- start start 2024-04-25T14:03:23.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc070e70 0x7f9dcc13a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 0x7f9dcc0b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc071280 con 0x7f9dcc156cd0 2024-04-25T14:03:23.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9dcc0b0e60 con 0x7f9dcc0b21e0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd32f0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9dcc071400 con 0x7f9dcc070e70 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40174/0 (socket says 172.21.15.150:40174) 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 -- 172.21.15.150:0/2845790106 learned_addr learned my addr 172.21.15.150:0/2845790106 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd108c640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc070e70 0x7f9dcc13a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd188d640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 0x7f9dcc0b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 -- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc070e70 msgr2=0x7f9dcc13a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc070e70 0x7f9dcc13a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 -- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 msgr2=0x7f9dcc0b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 0x7f9dcc0b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 -- 172.21.15.150:0/2845790106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dcc149860 con 0x7f9dcc156cd0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd188d640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 0x7f9dcc0b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dd088b640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c4b0 tx=0x7f9dbc00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.929+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc017070 con 0x7f9dcc156cd0 2024-04-25T14:03:23.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.930+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dbc005ce0 con 0x7f9dcc156cd0 2024-04-25T14:03:23.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.930+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc0072b0 con 0x7f9dcc156cd0 2024-04-25T14:03:23.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.930+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dcc0b2b30 con 0x7f9dcc156cd0 2024-04-25T14:03:23.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.930+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dcc15dfd0 con 0x7f9dcc156cd0 2024-04-25T14:03:23.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.931+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dcc076a40 con 0x7f9dcc156cd0 2024-04-25T14:03:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.931+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9dbc01a070 con 0x7f9dcc156cd0 2024-04-25T14:03:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.932+0000 7f9dc1ffb640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 0x7f9db00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.932+0000 7f9dd108c640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 0x7f9db00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.932+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7f9dbc094640 con 0x7f9dcc156cd0 2024-04-25T14:03:23.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.932+0000 7f9dd108c640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 0x7f9db00769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc074640 tx=0x7f9dc8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:23.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:23.936+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9dcc0b2b30 con 0x7f9dcc156cd0 2024-04-25T14:03:24.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.113+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dcc0b61d0 con 0x7f9dcc156cd0 2024-04-25T14:03:24.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.114+0000 7f9dc1ffb640 1 -- 172.21.15.150:0/2845790106 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) ==== 74+0+16955 (secure 0 0 0) 0x7f9dbc05e5f0 con 0x7f9dcc156cd0 2024-04-25T14:03:24.114 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:24.114 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:22.813136+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:24.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 msgr2=0x7f9db00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 0x7f9db00769c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc074640 tx=0x7f9dc8009190 comp rx=0 tx=0).stop 2024-04-25T14:03:24.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 msgr2=0x7f9dcc14ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c4b0 tx=0x7f9dbc00c970 comp rx=0 tx=0).stop 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 shutdown_connections 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9db00744b0 0x7f9db00769c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9dcc0b21e0 0x7f9dcc0b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9dcc156cd0 0x7f9dcc14ee20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 --2- 172.21.15.150:0/2845790106 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9dcc070e70 0x7f9dcc13a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 >> 172.21.15.150:0/2845790106 conn(0x7f9dcc0d1ae0 msgr2=0x7f9dcc06f140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 shutdown_connections 2024-04-25T14:03:24.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.116+0000 7f9dd32f0640 1 -- 172.21.15.150:0/2845790106 wait complete. 2024-04-25T14:03:24.128 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T14:03:24.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.240+0000 7fdb042af640 1 Processor -- start 2024-04-25T14:03:24.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.240+0000 7fdb042af640 1 -- start start 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.241+0000 7fdb042af640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdafc070820 0x7fdafc1609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.241+0000 7fdb042af640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 0x7fdafc160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.241+0000 7fdb042af640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdafc0cf300 con 0x7fdafc0d03a0 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.241+0000 7fdb042af640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fdafc0cff10 con 0x7fdafc070820 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.241+0000 7fdb042af640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdafc0cf5e0 con 0x7fdafc0cdca0 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdafc070820 0x7fdafc1609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fdafc070820 0x7fdafc1609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40184/0 (socket says 172.21.15.150:40184) 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0204b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdafc0d03a0 0x7fdafc0cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39130/0 (socket says 172.21.15.150:39130) 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 -- 172.21.15.150:0/26769162 learned_addr learned my addr 172.21.15.150:0/26769162 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:24.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0284c640 1 --2- 172.21.15.150:0/26769162 >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 0x7fdafc160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 -- 172.21.15.150:0/26769162 >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 msgr2=0x7fdafc160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 --2- 172.21.15.150:0/26769162 >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 0x7fdafc160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 -- 172.21.15.150:0/26769162 >> v1:172.21.15.150:6789/0 conn(0x7fdafc0d03a0 legacy=0x7fdafc0cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0184a640 1 -- 172.21.15.150:0/26769162 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc0cf8c0 con 0x7fdafc070820 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb01049640 1 -- 172.21.15.150:0/26769162 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1724012952 0 0) 0x7fdafc0cf300 con 0x7fdafc0d03a0 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.242+0000 7fdb0284c640 1 --2- 172.21.15.150:0/26769162 >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 0x7fdafc160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb0184a640 1 --2- 172.21.15.150:0/26769162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc1609e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0d0ac0 tx=0x7fdaec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5595c9b4e3854bb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb01049640 1 -- 172.21.15.150:0/26769162 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec037070 con 0x7fdafc070820 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb01049640 1 -- 172.21.15.150:0/26769162 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaec032440 con 0x7fdafc070820 2024-04-25T14:03:24.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb01049640 1 -- 172.21.15.150:0/26769162 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaec032810 con 0x7fdafc070820 2024-04-25T14:03:24.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb042af640 1 -- 172.21.15.150:0/26769162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 msgr2=0x7fdafc1609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.243+0000 7fdb042af640 1 --2- 172.21.15.150:0/26769162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc1609e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0d0ac0 tx=0x7fdaec02a270 comp rx=0 tx=0).stop 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 -- 172.21.15.150:0/26769162 shutdown_connections 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 --2- 172.21.15.150:0/26769162 >> v2:172.21.15.187:3300/0 conn(0x7fdafc0cdca0 0x7fdafc160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 --2- 172.21.15.150:0/26769162 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc1609e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 -- 172.21.15.150:0/26769162 >> 172.21.15.150:0/26769162 conn(0x7fdafc0ce390 msgr2=0x7fdafc0ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 -- 172.21.15.150:0/26769162 shutdown_connections 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.244+0000 7fdb042af640 1 -- 172.21.15.150:0/26769162 wait complete. 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.245+0000 7fdb042af640 1 Processor -- start 2024-04-25T14:03:24.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.245+0000 7fdb042af640 1 -- start start 2024-04-25T14:03:24.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 0x7fdafc0b6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 0x7fdafc1f30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc06f190 con 0x7fdafc070820 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdafc06f010 con 0x7fdafc0cdca0 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdafc06f310 con 0x7fdafc0d03a0 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40190/0 (socket says 172.21.15.150:40190) 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 -- 172.21.15.150:0/3745789399 learned_addr learned my addr 172.21.15.150:0/3745789399 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0284c640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 0x7fdafc1f30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0184a640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 0x7fdafc0b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 -- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 msgr2=0x7fdafc1f30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 0x7fdafc1f30c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 -- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 msgr2=0x7fdafc0b6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 0x7fdafc0b6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 -- 172.21.15.150:0/3745789399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdafc0b6de0 con 0x7fdafc070820 2024-04-25T14:03:24.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0284c640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 0x7fdafc1f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0184a640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 0x7fdafc0b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb0204b640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c410 tx=0x7fdaf800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80074e0 con 0x7fdafc070820 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaf80047e0 con 0x7fdafc070820 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdafc0b70c0 con 0x7fdafc070820 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.247+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaf8004bd0 con 0x7fdafc070820 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.248+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdafc0b1220 con 0x7fdafc070820 2024-04-25T14:03:24.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.248+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdafc0b1010 con 0x7fdafc070820 2024-04-25T14:03:24.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.249+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdaf8011070 con 0x7fdafc070820 2024-04-25T14:03:24.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.249+0000 7fdaf2ffd640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 0x7fdae00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:24.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.250+0000 7fdb0184a640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 0x7fdae00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:24.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.250+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6101+0+0 (secure 0 0 0) 0x7fdaf8094f90 con 0x7fdafc070820 2024-04-25T14:03:24.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.250+0000 7fdb0184a640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 0x7fdae00769f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdaec034000 tx=0x7fdaec02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:24.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.254+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaf805f590 con 0x7fdafc070820 2024-04-25T14:03:24.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:24.465+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdafc076a40 con 0x7fdafc070820 2024-04-25T14:03:25.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.210+0000 7fdaf2ffd640 1 -- 172.21.15.150:0/3745789399 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) ==== 118+0+0 (secure 0 0 0) 0x7fdaf805ee90 con 0x7fdafc070820 2024-04-25T14:03:25.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 msgr2=0x7fdae00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 0x7fdae00769f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fdaec034000 tx=0x7fdaec02e000 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 msgr2=0x7fdafc09e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800c410 tx=0x7fdaf800c8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 shutdown_connections 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdae00744e0 0x7fdae00769f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdafc0d03a0 0x7fdafc1f30c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdafc0cdca0 0x7fdafc0b6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 --2- 172.21.15.150:0/3745789399 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdafc070820 0x7fdafc09e8e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 >> 172.21.15.150:0/3745789399 conn(0x7fdafc0ce390 msgr2=0x7fdafc0cbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 shutdown_connections 2024-04-25T14:03:25.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.213+0000 7fdb042af640 1 -- 172.21.15.150:0/3745789399 wait complete. 2024-04-25T14:03:25.226 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T14:03:25.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T14:03:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.339+0000 7f4084b14640 1 Processor -- start 2024-04-25T14:03:25.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.339+0000 7f4084b14640 1 -- start start 2024-04-25T14:03:25.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f4084b14640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f408009b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f4084b14640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f408009b9e0 0x7f40800b3070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f4084b14640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408015ac60 con 0x7f4080160770 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f4084b14640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f408009bdf0 con 0x7f408009b9e0 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f4084b14640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4080160ce0 con 0x7f408015a850 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f407e59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4080160770 0x7f408006f2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39132/0 (socket says 172.21.15.150:39132) 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.340+0000 7f407e59c640 1 -- 172.21.15.150:0/3937530476 learned_addr learned my addr 172.21.15.150:0/3937530476 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407dd9b640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f408009b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942182850 0 0) 0x7f408015ac60 con 0x7f4080160770 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407ed9d640 1 --2- 172.21.15.150:0/3937530476 >> v2:172.21.15.187:3300/0 conn(0x7f408009b9e0 0x7f40800b3070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4068003520 con 0x7f4080160770 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 940058038 0 0) 0x7f4068003520 con 0x7f4080160770 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 msgr2=0x7f408009b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f408009b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 >> v2:172.21.15.187:3300/0 conn(0x7f408009b9e0 msgr2=0x7f40800b3070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 --2- 172.21.15.150:0/3937530476 >> v2:172.21.15.187:3300/0 conn(0x7f408009b9e0 0x7f40800b3070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080160fc0 con 0x7f4080160770 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f4070001670 con 0x7f4080160770 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 >> v1:172.21.15.150:6789/0 conn(0x7f4080160770 legacy=0x7f408006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f4068003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f4068004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4068004d40 0x7f4068005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f408009bdf0 con 0x7f40680039b0 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4080160ce0 con 0x7f4068004d40 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f406800b9c0 con 0x7f408015a850 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407e59c640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f4068003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407ed9d640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4068004d40 0x7f4068005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.341+0000 7f407dd9b640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f4068004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 -- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 msgr2=0x7f4068004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f408015a850 0x7f4068004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 -- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4068004d40 msgr2=0x7f4068005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4068004d40 0x7f4068005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 -- 172.21.15.150:0/3937530476 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4068003520 con 0x7f40680039b0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407e59c640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f4068003e20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4070001050 tx=0x7f4068005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9941c37b740d1cf2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007d70 con 0x7f40680039b0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4070005af0 con 0x7f40680039b0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.342+0000 7f407d59a640 1 -- 172.21.15.150:0/3937530476 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070007070 con 0x7f40680039b0 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 -- 172.21.15.150:0/3937530476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 msgr2=0x7f4068003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 --2- 172.21.15.150:0/3937530476 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f4068003e20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f4070001050 tx=0x7f4068005830 comp rx=0 tx=0).stop 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f407ed9d640 1 -- 172.21.15.150:0/3937530476 reap_dead start 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 -- 172.21.15.150:0/3937530476 shutdown_connections 2024-04-25T14:03:25.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 -- 172.21.15.150:0/3937530476 >> 172.21.15.150:0/3937530476 conn(0x7f40800b7680 msgr2=0x7f40800b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:25.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 -- 172.21.15.150:0/3937530476 shutdown_connections 2024-04-25T14:03:25.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.343+0000 7f4084b14640 1 -- 172.21.15.150:0/3937530476 wait complete. 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.345+0000 7f4084b14640 1 Processor -- start 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.345+0000 7f4084b14640 1 -- start start 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4080157990 0x7f4080157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4080146d80 0x7f4080147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f408009b4a0 con 0x7f40680039b0 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4080161580 con 0x7f4080146d80 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.346+0000 7f4084b14640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f408009b620 con 0x7f4080157990 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40204/0 (socket says 172.21.15.150:40204) 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 -- 172.21.15.150:0/3659154480 learned_addr learned my addr 172.21.15.150:0/3659154480 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:25.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407ed9d640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4080146d80 0x7f4080147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407dd9b640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4080157990 0x7f4080157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 -- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4080157990 msgr2=0x7f4080157da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4080157990 0x7f4080157da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 -- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4080146d80 msgr2=0x7f4080147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4080146d80 0x7f4080147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 -- 172.21.15.150:0/3659154480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40801582e0 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f407e59c640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f407002add0 tx=0x7f4070000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4070001630 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4070042070 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40700072b0 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40801476d0 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.347+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4080145130 con 0x7f40680039b0 2024-04-25T14:03:25.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.348+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4080158730 con 0x7f40680039b0 2024-04-25T14:03:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.349+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4070007450 con 0x7f40680039b0 2024-04-25T14:03:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.349+0000 7f406effd640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 0x7f40600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.349+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f40700b54b0 con 0x7f40680039b0 2024-04-25T14:03:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.349+0000 7f407dd9b640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 0x7f40600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:25.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.350+0000 7f407dd9b640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 0x7f40600769c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4080144bc0 tx=0x7f4074009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:25.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.355+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4080158730 con 0x7f40680039b0 2024-04-25T14:03:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:25.570+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f4080076a40 con 0x7f40680039b0 2024-04-25T14:03:26.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.482+0000 7f406effd640 1 -- 172.21.15.150:0/3659154480 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) ==== 122+0+0 (secure 0 0 0) 0x7f40801476d0 con 0x7f40680039b0 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 msgr2=0x7f40600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 0x7f40600769c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4080144bc0 tx=0x7f4074009190 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 msgr2=0x7f408014aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f407002add0 tx=0x7f4070000fe0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 shutdown_connections 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f40600744b0 0x7f40600769c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4080146d80 0x7f4080147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4080157990 0x7f4080157da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 --2- 172.21.15.150:0/3659154480 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f40680039b0 0x7f408014aa90 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.487 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.484+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 >> 172.21.15.150:0/3659154480 conn(0x7f40800b7680 msgr2=0x7f40800b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.485+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 shutdown_connections 2024-04-25T14:03:26.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.485+0000 7f4084b14640 1 -- 172.21.15.150:0/3659154480 wait complete. 2024-04-25T14:03:26.497 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T14:03:26.498 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 Processor -- start 2024-04-25T14:03:26.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 -- start start 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68f406d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f406c570 con 0x7f68f406d830 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68f41605e0 con 0x7f68f41601d0 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.610+0000 7f68f98aa640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f68f40d5980 con 0x7f68f40b9070 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f37fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f68f406d830 0x7f68f4163ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39146/0 (socket says 172.21.15.150:39146) 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f37fe640 1 -- 172.21.15.150:0/3747555593 learned_addr learned my addr 172.21.15.150:0/3747555593 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68ebfff640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68f406d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 722693938 0 0) 0x7f68f406c570 con 0x7f68f406d830 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68e0003520 con 0x7f68f406d830 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2812218101 0 0) 0x7f68f41605e0 con 0x7f68f41601d0 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f68f406c570 con 0x7f68f41601d0 2024-04-25T14:03:26.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1667708833 0 0) 0x7f68e0003520 con 0x7f68f406d830 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 msgr2=0x7f68f406d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68f406d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 >> v1:172.21.15.187:6789/0 conn(0x7f68f41601d0 legacy=0x7f68f40d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f406c810 con 0x7f68f406d830 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f68ec0019f0 con 0x7f68f406d830 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 >> v1:172.21.15.150:6789/0 conn(0x7f68f406d830 legacy=0x7f68f4163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e0003b70 0x7f68e0003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68e0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68e0004d40 0x7f68e0005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40d5980 con 0x7f68e0004d40 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f68e000bbb0 con 0x7f68e0003b70 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f68e000bdb0 con 0x7f68f40b9070 2024-04-25T14:03:26.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f37fe640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68e0004d40 0x7f68e0005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68f2ffd640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e0003b70 0x7f68e0003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.611+0000 7f68ebfff640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68e0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 -- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 msgr2=0x7f68e0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68f40b9070 0x7f68e0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 -- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e0003b70 msgr2=0x7f68e0003fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68e0003b70 0x7f68e0003fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 -- 172.21.15.150:0/3747555593 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e0003520 con 0x7f68e0004d40 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.612+0000 7f68f37fe640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68e0004d40 0x7f68e0005150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00a7f0 tx=0x7f68e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1366d021b839e562 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec00a410 con 0x7f68e0004d40 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec02f070 con 0x7f68e0004d40 2024-04-25T14:03:26.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f27fc640 1 -- 172.21.15.150:0/3747555593 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec02bc50 con 0x7f68e0004d40 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 -- 172.21.15.150:0/3747555593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68e0004d40 msgr2=0x7f68e0005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 --2- 172.21.15.150:0/3747555593 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68e0004d40 0x7f68e0005150 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f68ec00a7f0 tx=0x7f68e0005830 comp rx=0 tx=0).stop 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f37fe640 1 -- 172.21.15.150:0/3747555593 reap_dead start 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 -- 172.21.15.150:0/3747555593 shutdown_connections 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 -- 172.21.15.150:0/3747555593 >> 172.21.15.150:0/3747555593 conn(0x7f68f40d5d90 msgr2=0x7f68f40d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:26.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 -- 172.21.15.150:0/3747555593 shutdown_connections 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.614+0000 7f68f98aa640 1 -- 172.21.15.150:0/3747555593 wait complete. 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.616+0000 7f68f98aa640 1 Processor -- start 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.616+0000 7f68f98aa640 1 -- start start 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68e0004d40 0x7f68f41f9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 0x7f68f41619c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f68f40b9480 con 0x7f68f4154b10 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f68f406cdd0 con 0x7f68f40cf870 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f68f40b9600 con 0x7f68e0004d40 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40230/0 (socket says 172.21.15.150:40230) 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 -- 172.21.15.150:0/2679460940 learned_addr learned my addr 172.21.15.150:0/2679460940 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f2ffd640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68e0004d40 0x7f68f41f9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68ebfff640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 0x7f68f41619c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 -- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68e0004d40 msgr2=0x7f68f41f9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68e0004d40 0x7f68f41f9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 -- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 msgr2=0x7f68f41619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 0x7f68f41619c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 -- 172.21.15.150:0/2679460940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68f41607e0 con 0x7f68f4154b10 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68ebfff640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 0x7f68f41619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f37fe640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f68ec02b4c0 tx=0x7f68f406c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec001f80 con 0x7f68f4154b10 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68ec02bc20 con 0x7f68f4154b10 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68ec034470 con 0x7f68f4154b10 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68f41fa6b0 con 0x7f68f4154b10 2024-04-25T14:03:26.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.617+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68f41fac70 con 0x7f68f4154b10 2024-04-25T14:03:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.619+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68f4076a40 con 0x7f68f4154b10 2024-04-25T14:03:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.619+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f68ec034610 con 0x7f68f4154b10 2024-04-25T14:03:26.621 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.621+0000 7f68f0ff9640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 0x7f68d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.621+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f68ec0b53b0 con 0x7f68f4154b10 2024-04-25T14:03:26.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.621+0000 7f68f2ffd640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 0x7f68d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.621+0000 7f68f2ffd640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 0x7f68d40769c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f68f408df30 tx=0x7f68e4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.625+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68ec07fb00 con 0x7f68f4154b10 2024-04-25T14:03:26.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.838+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f68f40d3fb0 con 0x7f68f4154b10 2024-04-25T14:03:26.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.838+0000 7f68f0ff9640 1 -- 172.21.15.150:0/2679460940 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5184 (secure 0 0 0) 0x7f68ec07f400 con 0x7f68f4154b10 2024-04-25T14:03:26.838 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:26.838 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/21568960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":21568960},{"type":"v1","addr":"172.21.15.187:6834","nonce":21568960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5176,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/265568101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":265568101},{"type":"v1","addr":"172.21.15.150:6837","nonce":265568101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5185,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1241446695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":1241446695},{"type":"v1","addr":"172.21.15.187:6835","nonce":1241446695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:25.572204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5349},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5349":{"gid":5349,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6836/3196344346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3196344346},{"type":"v1","addr":"172.21.15.150:6836","nonce":3196344346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5349,"qdb_cluster":[5349]},"id":2}]} 2024-04-25T14:03:26.839 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 msgr2=0x7f68d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 0x7f68d40769c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f68f408df30 tx=0x7f68e4009040 comp rx=0 tx=0).stop 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 msgr2=0x7f68f4160fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f68ec02b4c0 tx=0x7f68f406c9e0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 shutdown_connections 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68d40744b0 0x7f68d40769c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.840 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f68f4154b10 0x7f68f4160fa0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f68f40cf870 0x7f68f41619c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.840+0000 7f68f98aa640 1 --2- 172.21.15.150:0/2679460940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f68e0004d40 0x7f68f41f9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.841+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 >> 172.21.15.150:0/2679460940 conn(0x7f68f40d5d90 msgr2=0x7f68f40d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:26.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.841+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 shutdown_connections 2024-04-25T14:03:26.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.841+0000 7f68f98aa640 1 -- 172.21.15.150:0/2679460940 wait complete. 2024-04-25T14:03:26.851 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:03:26.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.960+0000 7f3b7b3e0640 1 Processor -- start 2024-04-25T14:03:26.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.960+0000 7f3b7b3e0640 1 -- start start 2024-04-25T14:03:26.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.961+0000 7f3b7b3e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7415d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.961+0000 7f3b7b3e0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 0x7f3b7406ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.961+0000 7f3b7b3e0640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b7406d750 con 0x7f3b7406d340 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.961+0000 7f3b7b3e0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3b74070250 con 0x7f3b74070e70 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.961+0000 7f3b7b3e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b740afe00 con 0x7f3b74156cd0 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7997d640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f3b7406d340 0x7f3b74160180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:39150/0 (socket says 172.21.15.150:39150) 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7997d640 1 -- 172.21.15.150:0/1881131916 learned_addr learned my addr 172.21.15.150:0/1881131916 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7917c640 1 --2- 172.21.15.150:0/1881131916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7415d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7897b640 1 --2- 172.21.15.150:0/1881131916 >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 0x7f3b7406ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3895467593 0 0) 0x7f3b6c001490 con 0x7f3b7406d340 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b60003520 con 0x7f3b7406d340 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7917c640 1 -- 172.21.15.150:0/1881131916 >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 msgr2=0x7f3b7406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7917c640 1 --2- 172.21.15.150:0/1881131916 >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 0x7f3b7406ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1505264764 0 0) 0x7f3b60003520 con 0x7f3b7406d340 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7917c640 1 -- 172.21.15.150:0/1881131916 >> v1:172.21.15.150:6789/0 conn(0x7f3b7406d340 legacy=0x7f3b74160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7917c640 1 -- 172.21.15.150:0/1881131916 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b740b00a0 con 0x7f3b74156cd0 2024-04-25T14:03:26.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.962+0000 7f3b7897b640 1 --2- 172.21.15.150:0/1881131916 >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 0x7f3b7406ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.963+0000 7f3b7917c640 1 --2- 172.21.15.150:0/1881131916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7415d7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000a940 tx=0x7f3b7002b4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=836845b157d00a7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.963+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6c001bc0 con 0x7f3b74156cd0 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.963+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b70030510 con 0x7f3b74156cd0 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.963+0000 7f3b6bfff640 1 -- 172.21.15.150:0/1881131916 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b7000b040 con 0x7f3b74156cd0 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/1881131916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 msgr2=0x7f3b7415d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/1881131916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7415d7a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3b7000a940 tx=0x7f3b7002b4d0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/1881131916 shutdown_connections 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/1881131916 >> v2:172.21.15.187:3300/0 conn(0x7f3b74070e70 0x7f3b7406ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/1881131916 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7415d7a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/1881131916 >> 172.21.15.150:0/1881131916 conn(0x7f3b740d1ae0 msgr2=0x7f3b740d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/1881131916 shutdown_connections 2024-04-25T14:03:26.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.964+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/1881131916 wait complete. 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.966+0000 7f3b7b3e0640 1 Processor -- start 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.966+0000 7f3b7b3e0640 1 -- start start 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b7406d340 0x7f3b74150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3b740b0940 con 0x7f3b74070e70 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3b740b0660 con 0x7f3b7406d340 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7b3e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3b740b0e60 con 0x7f3b74156cd0 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40236/0 (socket says 172.21.15.150:40236) 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 -- 172.21.15.150:0/362670553 learned_addr learned my addr 172.21.15.150:0/362670553 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7917c640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b7406d340 0x7f3b74150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7997d640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 -- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 msgr2=0x7f3b7414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 -- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b7406d340 msgr2=0x7f3b74150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b7406d340 0x7f3b74150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7897b640 1 -- 172.21.15.150:0/362670553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b7413a640 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.967+0000 7f3b7997d640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b7897b640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400e990 tx=0x7f3b6400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b64017070 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b6400c9e0 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b741f6210 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b6400cdb0 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.968+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b741f67d0 con 0x7f3b74070e70 2024-04-25T14:03:26.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.969+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b74076a40 con 0x7f3b74070e70 2024-04-25T14:03:26.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.970+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3b6401a070 con 0x7f3b74070e70 2024-04-25T14:03:26.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.970+0000 7f3b69ffb640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 0x7f3b580769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:26.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.970+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3b64095590 con 0x7f3b74070e70 2024-04-25T14:03:26.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.970+0000 7f3b7917c640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 0x7f3b580769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:26.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.971+0000 7f3b7917c640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 0x7f3b580769f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3b70035000 tx=0x7f3b70030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:26.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:26.975+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b6405fc20 con 0x7f3b74070e70 2024-04-25T14:03:27.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.152+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b740b61d0 con 0x7f3b74070e70 2024-04-25T14:03:27.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.153+0000 7f3b69ffb640 1 -- 172.21.15.150:0/362670553 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16942 (secure 0 0 0) 0x7f3b6405f520 con 0x7f3b74070e70 2024-04-25T14:03:27.153 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:27.153 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":38,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:24.466671+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:03:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 msgr2=0x7f3b580769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 0x7f3b580769f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3b70035000 tx=0x7f3b70030000 comp rx=0 tx=0).stop 2024-04-25T14:03:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 msgr2=0x7f3b7413a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f3b6400e990 tx=0x7f3b6400ee50 comp rx=0 tx=0).stop 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 shutdown_connections 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3b580744e0 0x7f3b580769f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3b74156cd0 0x7f3b7414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3b74070e70 0x7f3b7413a100 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 --2- 172.21.15.150:0/362670553 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3b7406d340 0x7f3b74150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 >> 172.21.15.150:0/362670553 conn(0x7f3b740d1ae0 msgr2=0x7f3b7414fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 shutdown_connections 2024-04-25T14:03:27.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.156+0000 7f3b7b3e0640 1 -- 172.21.15.150:0/362670553 wait complete. 2024-04-25T14:03:27.166 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T14:03:27.167 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:03:27.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.275+0000 7fc3f4230640 1 Processor -- start 2024-04-25T14:03:27.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.275+0000 7fc3f4230640 1 -- start start 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f4230640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f4230640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f4230640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3ec0afe00 con 0x7fc3ec1606c0 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f4230640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ec0b00a0 con 0x7fc3ec156cd0 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f4230640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec0b0380 con 0x7fc3ec070e70 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f1fcc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f1fcc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40252/0 (socket says 172.21.15.150:40252) 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.276+0000 7fc3f1fcc640 1 -- 172.21.15.150:0/4087385383 learned_addr learned my addr 172.21.15.150:0/4087385383 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f17cb640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f1fcc640 1 -- 172.21.15.150:0/4087385383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 msgr2=0x7fc3ec160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:27.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f1fcc640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f1fcc640 1 -- 172.21.15.150:0/4087385383 >> v1:172.21.15.187:6789/0 conn(0x7fc3ec1606c0 legacy=0x7fc3ec06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:27.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f1fcc640 1 -- 172.21.15.150:0/4087385383 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec0b0660 con 0x7fc3ec156cd0 2024-04-25T14:03:27.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f17cb640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:27.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f1fcc640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec074690 tx=0x7fc3e802a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622823b1e9dfd2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.277+0000 7fc3f0fca640 1 -- 172.21.15.150:0/4087385383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8037070 con 0x7fc3ec156cd0 2024-04-25T14:03:27.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f0fca640 1 -- 172.21.15.150:0/4087385383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e8032440 con 0x7fc3ec156cd0 2024-04-25T14:03:27.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f0fca640 1 -- 172.21.15.150:0/4087385383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8032810 con 0x7fc3ec156cd0 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 -- 172.21.15.150:0/4087385383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec156cd0 msgr2=0x7fc3ec15d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec074690 tx=0x7fc3e802a200 comp rx=0 tx=0).stop 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 -- 172.21.15.150:0/4087385383 shutdown_connections 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 --2- 172.21.15.150:0/4087385383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec15d7a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 -- 172.21.15.150:0/4087385383 >> 172.21.15.150:0/4087385383 conn(0x7fc3ec0d1ae0 msgr2=0x7fc3ec0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 -- 172.21.15.150:0/4087385383 shutdown_connections 2024-04-25T14:03:27.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.278+0000 7fc3f4230640 1 -- 172.21.15.150:0/4087385383 wait complete. 2024-04-25T14:03:27.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.280+0000 7fc3f4230640 1 Processor -- start 2024-04-25T14:03:27.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.280+0000 7fc3f4230640 1 -- start start 2024-04-25T14:03:27.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec09ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec071280 con 0x7fc3ec1606c0 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec0b0e60 con 0x7fc3ec156cd0 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f4230640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec071400 con 0x7fc3ec070e70 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f27cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.281+0000 7fc3f27cd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:40268/0 (socket says 172.21.15.150:40268) 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 -- 172.21.15.150:0/1601536711 learned_addr learned my addr 172.21.15.150:0/1601536711 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f17cb640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f1fcc640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec09ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 -- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 msgr2=0x7fc3ec09ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec09ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 -- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3ec156cd0 msgr2=0x7fc3ec155870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec155870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 -- 172.21.15.150:0/1601536711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec150e70 con 0x7fc3ec1606c0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f27cd640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400ba10 tx=0x7fc3e400bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4019070 con 0x7fc3ec1606c0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e400cb40 con 0x7fc3ec1606c0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4014440 con 0x7fc3ec1606c0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec14a0a0 con 0x7fc3ec1606c0 2024-04-25T14:03:27.283 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.282+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec1513c0 con 0x7fc3ec1606c0 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.283+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ec076a40 con 0x7fc3ec1606c0 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.284+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc3e401c070 con 0x7fc3ec1606c0 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.284+0000 7fc3e27fc640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 0x7fc3d00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.285+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fc3e4094ea0 con 0x7fc3ec1606c0 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.285+0000 7fc3f1fcc640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 0x7fc3d00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.285+0000 7fc3f1fcc640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 0x7fc3d00769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8034000 tx=0x7fc3e802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.289+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc3e405ee50 con 0x7fc3ec1606c0 2024-04-25T14:03:27.429 INFO:tasks.ceph.mds.c.smithi150.stderr:2024-04-25T14:03:27.430+0000 7f3bb09da640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:03:27.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.459+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc3ec1425c0 con 0x7fc3d00744e0 2024-04-25T14:03:27.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.465+0000 7fc3e27fc640 1 -- 172.21.15.150:0/1601536711 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240060 (secure 0 0 0) 0x7fc3e807c9d0 con 0x7fc3d00744e0 2024-04-25T14:03:27.467 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:27.468 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:03:27.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 msgr2=0x7fc3d00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 0x7fc3d00769f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8034000 tx=0x7fc3e802e000 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 msgr2=0x7fc3ec1518e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400ba10 tx=0x7fc3e400bed0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 shutdown_connections 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc3d00744e0 0x7fc3d00769f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc3ec1606c0 0x7fc3ec1518e0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc3ec156cd0 0x7fc3ec155870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 --2- 172.21.15.150:0/1601536711 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc3ec070e70 0x7fc3ec09ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 >> 172.21.15.150:0/1601536711 conn(0x7fc3ec0d1ae0 msgr2=0x7fc3ec1505b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 shutdown_connections 2024-04-25T14:03:27.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.468+0000 7fc3f4230640 1 -- 172.21.15.150:0/1601536711 wait complete. 2024-04-25T14:03:27.479 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-25T14:03:26.603503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220416,"kb_used_data":4212,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754304,"statfs":{"total":773094113280,"available":772868407296,"internally_reserved":0,"allocated":4313088,"data_stored":2493353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160575"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448728+0000","last_change":"2024-04-25T14:03:21.448728+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448728+0000","last_undegraded":"2024-04-25T14:03:21.448728+0000","last_fullsized":"2024-04-25T14:03:21.448728+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451883+0000","last_change":"2024-04-25T14:03:21.451883+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451883+0000","last_undegraded":"2024-04-25T14:03:21.451883+0000","last_fullsized":"2024-04-25T14:03:21.451883+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451477+0000","last_change":"2024-04-25T14:03:21.451477+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451477+0000","last_undegraded":"2024-04-25T14:03:21.451477+0000","last_fullsized":"2024-04-25T14:03:21.451477+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449012+0000","last_change":"2024-04-25T14:03:21.449012+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449012+0000","last_undegraded":"2024-04-25T14:03:21.449012+0000","last_fullsized":"2024-04-25T14:03:21.449012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444412+0000","last_change":"2024-04-25T14:03:21.444412+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444412+0000","last_undegraded":"2024-04-25T14:03:21.444412+0000","last_fullsized":"2024-04-25T14:03:21.444412+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451346+0000","last_change":"2024-04-25T14:03:21.451346+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451346+0000","last_undegraded":"2024-04-25T14:03:21.451346+0000","last_fullsized":"2024-04-25T14:03:21.451346+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445316+0000","last_change":"2024-04-25T14:03:21.445316+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445316+0000","last_undegraded":"2024-04-25T14:03:21.445316+0000","last_fullsized":"2024-04-25T14:03:21.445316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448560+0000","last_change":"2024-04-25T14:03:21.448560+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448560+0000","last_undegraded":"2024-04-25T14:03:21.448560+0000","last_fullsized":"2024-04-25T14:03:21.448560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449111+0000","last_change":"2024-04-25T14:03:21.449111+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449111+0000","last_undegraded":"2024-04-25T14:03:21.449111+0000","last_fullsized":"2024-04-25T14:03:21.449111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448969+0000","last_change":"2024-04-25T14:03:21.448969+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448969+0000","last_undegraded":"2024-04-25T14:03:21.448969+0000","last_fullsized":"2024-04-25T14:03:21.448969+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449004+0000","last_change":"2024-04-25T14:03:21.449004+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449004+0000","last_undegraded":"2024-04-25T14:03:21.449004+0000","last_fullsized":"2024-04-25T14:03:21.449004+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451914+0000","last_change":"2024-04-25T14:03:21.451914+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451914+0000","last_undegraded":"2024-04-25T14:03:21.451914+0000","last_fullsized":"2024-04-25T14:03:21.451914+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448722+0000","last_change":"2024-04-25T14:03:21.448722+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448722+0000","last_undegraded":"2024-04-25T14:03:21.448722+0000","last_fullsized":"2024-04-25T14:03:21.448722+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444308+0000","last_change":"2024-04-25T14:03:21.444308+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444308+0000","last_undegraded":"2024-04-25T14:03:21.444308+0000","last_fullsized":"2024-04-25T14:03:21.444308+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449771+0000","last_change":"2024-04-25T14:03:21.449771+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449771+0000","last_undegraded":"2024-04-25T14:03:21.449771+0000","last_fullsized":"2024-04-25T14:03:21.449771+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445677+0000","last_change":"2024-04-25T14:03:21.445677+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445677+0000","last_undegraded":"2024-04-25T14:03:21.445677+0000","last_fullsized":"2024-04-25T14:03:21.445677+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449716+0000","last_change":"2024-04-25T14:03:21.449716+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449716+0000","last_undegraded":"2024-04-25T14:03:21.449716+0000","last_fullsized":"2024-04-25T14:03:21.449716+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451940+0000","last_change":"2024-04-25T14:03:21.451940+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451940+0000","last_undegraded":"2024-04-25T14:03:21.451940+0000","last_fullsized":"2024-04-25T14:03:21.451940+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449662+0000","last_change":"2024-04-25T14:03:21.449662+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449662+0000","last_undegraded":"2024-04-25T14:03:21.449662+0000","last_fullsized":"2024-04-25T14:03:21.449662+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449170+0000","last_change":"2024-04-25T14:03:21.449170+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449170+0000","last_undegraded":"2024-04-25T14:03:21.449170+0000","last_fullsized":"2024-04-25T14:03:21.449170+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448619+0000","last_change":"2024-04-25T14:03:21.448619+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448619+0000","last_undegraded":"2024-04-25T14:03:21.448619+0000","last_fullsized":"2024-04-25T14:03:21.448619+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449266+0000","last_change":"2024-04-25T14:03:21.449266+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449266+0000","last_undegraded":"2024-04-25T14:03:21.449266+0000","last_fullsized":"2024-04-25T14:03:21.449266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451353+0000","last_change":"2024-04-25T14:03:21.451353+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451353+0000","last_undegraded":"2024-04-25T14:03:21.451353+0000","last_fullsized":"2024-04-25T14:03:21.451353+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451912+0000","last_change":"2024-04-25T14:03:21.451912+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451912+0000","last_undegraded":"2024-04-25T14:03:21.451912+0000","last_fullsized":"2024-04-25T14:03:21.451912+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448729+0000","last_change":"2024-04-25T14:03:21.448729+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448729+0000","last_undegraded":"2024-04-25T14:03:21.448729+0000","last_fullsized":"2024-04-25T14:03:21.448729+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445331+0000","last_change":"2024-04-25T14:03:21.445331+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445331+0000","last_undegraded":"2024-04-25T14:03:21.445331+0000","last_fullsized":"2024-04-25T14:03:21.445331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449046+0000","last_change":"2024-04-25T14:03:21.449046+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449046+0000","last_undegraded":"2024-04-25T14:03:21.449046+0000","last_fullsized":"2024-04-25T14:03:21.449046+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444727+0000","last_change":"2024-04-25T14:03:21.444727+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444727+0000","last_undegraded":"2024-04-25T14:03:21.444727+0000","last_fullsized":"2024-04-25T14:03:21.444727+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":140,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-25T14:03:21.445636+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:03:21.445636+0000","last_peered":"2024-04-25T14:03:21.445636+0000","last_clean":"2024-04-25T14:03:21.445636+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:03:21.445636+0000","last_undegraded":"2024-04-25T14:03:21.445636+0000","last_fullsized":"2024-04-25T14:03:21.445636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":759878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:03:27.483 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:03:27.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.628+0000 7f3d3b91c640 1 Processor -- start 2024-04-25T14:03:27.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.629+0000 7f3d3b91c640 1 -- start start 2024-04-25T14:03:27.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d3b91c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d3b91c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 0x7f3d340b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d3b91c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d340b5cf0 con 0x7f3d340b2b10 2024-04-25T14:03:27.629 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d3b91c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3d340b7db0 con 0x7f3d340d2e90 2024-04-25T14:03:27.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d3b91c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3d34164690 con 0x7f3d340b53a0 2024-04-25T14:03:27.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d39eb9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 0x7f3d340b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d38eb7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d38eb7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:48642/0 (socket says 172.21.15.150:48642) 2024-04-25T14:03:27.630 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.630+0000 7f3d38eb7640 1 -- 172.21.15.150:0/2527271998 learned_addr learned my addr 172.21.15.150:0/2527271998 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d39eb9640 1 -- 172.21.15.150:0/2527271998 >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 msgr2=0x7f3d34160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d39eb9640 1 --2- 172.21.15.150:0/2527271998 >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d39eb9640 1 -- 172.21.15.150:0/2527271998 >> v1:172.21.15.187:6789/0 conn(0x7f3d340b2b10 legacy=0x7f3d340cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d39eb9640 1 -- 172.21.15.150:0/2527271998 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d34164bc0 con 0x7f3d340b53a0 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d38eb7640 1 --2- 172.21.15.150:0/2527271998 >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.631+0000 7f3d39eb9640 1 --2- 172.21.15.150:0/2527271998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 0x7f3d340b57b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0097a0 tx=0x7f3d2c02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9b0ea6b2d18c3356 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d2bfff640 1 -- 172.21.15.150:0/2527271998 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c037070 con 0x7f3d340b53a0 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d2bfff640 1 -- 172.21.15.150:0/2527271998 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c02adb0 con 0x7f3d340b53a0 2024-04-25T14:03:27.631 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d2bfff640 1 -- 172.21.15.150:0/2527271998 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c032660 con 0x7f3d340b53a0 2024-04-25T14:03:27.632 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 -- 172.21.15.150:0/2527271998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 msgr2=0x7f3d340b57b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/2527271998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 0x7f3d340b57b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0097a0 tx=0x7f3d2c02a060 comp rx=0 tx=0).stop 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 -- 172.21.15.150:0/2527271998 shutdown_connections 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/2527271998 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340b53a0 0x7f3d340b57b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/2527271998 >> v2:172.21.15.187:3300/0 conn(0x7f3d340d2e90 0x7f3d34160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.632+0000 7f3d3b91c640 1 -- 172.21.15.150:0/2527271998 >> 172.21.15.150:0/2527271998 conn(0x7f3d340d03a0 msgr2=0x7f3d340d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.633+0000 7f3d3b91c640 1 -- 172.21.15.150:0/2527271998 shutdown_connections 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.633+0000 7f3d3b91c640 1 -- 172.21.15.150:0/2527271998 wait complete. 2024-04-25T14:03:27.633 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.634+0000 7f3d3b91c640 1 Processor -- start 2024-04-25T14:03:27.634 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.634+0000 7f3d3b91c640 1 -- start start 2024-04-25T14:03:27.636 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 0x7f3d341eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340d2e90 0x7f3d341eaf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3d341648e0 con 0x7f3d3414cb30 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3d340b8350 con 0x7f3d340b53a0 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d3b91c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3d340b7fe0 con 0x7f3d340d2e90 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45134/0 (socket says 172.21.15.150:45134) 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 -- 172.21.15.150:0/3746931326 learned_addr learned my addr 172.21.15.150:0/3746931326 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d38eb7640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340d2e90 0x7f3d341eaf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d396b8640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 0x7f3d341eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 -- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340d2e90 msgr2=0x7f3d341eaf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340d2e90 0x7f3d341eaf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 -- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 msgr2=0x7f3d341eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 0x7f3d341eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 -- 172.21.15.150:0/3746931326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3406e0b0 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d396b8640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 0x7f3d341eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d39eb9640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0098c0 tx=0x7f3d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.636+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c041020 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.637+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2c03c670 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.637+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2c037040 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.637+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3406e390 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.637+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d3406e950 con 0x7f3d3414cb30 2024-04-25T14:03:27.638 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.638+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d34076a40 con 0x7f3d3414cb30 2024-04-25T14:03:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.638+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3d2c044020 con 0x7f3d3414cb30 2024-04-25T14:03:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.639+0000 7f3d29ffb640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 0x7f3d180769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.639 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.639+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3d2c0b5d70 con 0x7f3d3414cb30 2024-04-25T14:03:27.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.639+0000 7f3d396b8640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 0x7f3d180769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.639+0000 7f3d396b8640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 0x7f3d180769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d30001d60 tx=0x7f3d30009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.643+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d2c04b020 con 0x7f3d3414cb30 2024-04-25T14:03:27.813 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.814+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d34068c30 con 0x7f3d180744e0 2024-04-25T14:03:27.819 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.820+0000 7f3d29ffb640 1 -- 172.21.15.150:0/3746931326 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240060 (secure 0 0 0) 0x7f3d34068c30 con 0x7f3d180744e0 2024-04-25T14:03:27.821 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:27.823 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:03:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 msgr2=0x7f3d180769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 0x7f3d180769f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3d30001d60 tx=0x7f3d30009040 comp rx=0 tx=0).stop 2024-04-25T14:03:27.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 msgr2=0x7f3d3406f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3d2c0098c0 tx=0x7f3d2c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 shutdown_connections 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3d180744e0 0x7f3d180769f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3d3414cb30 0x7f3d3406f150 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3d340d2e90 0x7f3d341eaf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 --2- 172.21.15.150:0/3746931326 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3d340b53a0 0x7f3d341eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 >> 172.21.15.150:0/3746931326 conn(0x7f3d340d03a0 msgr2=0x7f3d340b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 shutdown_connections 2024-04-25T14:03:27.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.823+0000 7f3d3b91c640 1 -- 172.21.15.150:0/3746931326 wait complete. 2024-04-25T14:03:27.833 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-25T14:03:26.603503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220416,"kb_used_data":4212,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754304,"statfs":{"total":773094113280,"available":772868407296,"internally_reserved":0,"allocated":4313088,"data_stored":2493353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160575"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448728+0000","last_change":"2024-04-25T14:03:21.448728+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448728+0000","last_undegraded":"2024-04-25T14:03:21.448728+0000","last_fullsized":"2024-04-25T14:03:21.448728+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451883+0000","last_change":"2024-04-25T14:03:21.451883+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451883+0000","last_undegraded":"2024-04-25T14:03:21.451883+0000","last_fullsized":"2024-04-25T14:03:21.451883+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451477+0000","last_change":"2024-04-25T14:03:21.451477+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451477+0000","last_undegraded":"2024-04-25T14:03:21.451477+0000","last_fullsized":"2024-04-25T14:03:21.451477+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449012+0000","last_change":"2024-04-25T14:03:21.449012+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449012+0000","last_undegraded":"2024-04-25T14:03:21.449012+0000","last_fullsized":"2024-04-25T14:03:21.449012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444412+0000","last_change":"2024-04-25T14:03:21.444412+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444412+0000","last_undegraded":"2024-04-25T14:03:21.444412+0000","last_fullsized":"2024-04-25T14:03:21.444412+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451346+0000","last_change":"2024-04-25T14:03:21.451346+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451346+0000","last_undegraded":"2024-04-25T14:03:21.451346+0000","last_fullsized":"2024-04-25T14:03:21.451346+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445316+0000","last_change":"2024-04-25T14:03:21.445316+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445316+0000","last_undegraded":"2024-04-25T14:03:21.445316+0000","last_fullsized":"2024-04-25T14:03:21.445316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448560+0000","last_change":"2024-04-25T14:03:21.448560+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448560+0000","last_undegraded":"2024-04-25T14:03:21.448560+0000","last_fullsized":"2024-04-25T14:03:21.448560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449111+0000","last_change":"2024-04-25T14:03:21.449111+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449111+0000","last_undegraded":"2024-04-25T14:03:21.449111+0000","last_fullsized":"2024-04-25T14:03:21.449111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448969+0000","last_change":"2024-04-25T14:03:21.448969+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448969+0000","last_undegraded":"2024-04-25T14:03:21.448969+0000","last_fullsized":"2024-04-25T14:03:21.448969+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449004+0000","last_change":"2024-04-25T14:03:21.449004+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449004+0000","last_undegraded":"2024-04-25T14:03:21.449004+0000","last_fullsized":"2024-04-25T14:03:21.449004+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451914+0000","last_change":"2024-04-25T14:03:21.451914+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451914+0000","last_undegraded":"2024-04-25T14:03:21.451914+0000","last_fullsized":"2024-04-25T14:03:21.451914+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448722+0000","last_change":"2024-04-25T14:03:21.448722+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448722+0000","last_undegraded":"2024-04-25T14:03:21.448722+0000","last_fullsized":"2024-04-25T14:03:21.448722+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444308+0000","last_change":"2024-04-25T14:03:21.444308+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444308+0000","last_undegraded":"2024-04-25T14:03:21.444308+0000","last_fullsized":"2024-04-25T14:03:21.444308+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449771+0000","last_change":"2024-04-25T14:03:21.449771+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449771+0000","last_undegraded":"2024-04-25T14:03:21.449771+0000","last_fullsized":"2024-04-25T14:03:21.449771+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445677+0000","last_change":"2024-04-25T14:03:21.445677+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445677+0000","last_undegraded":"2024-04-25T14:03:21.445677+0000","last_fullsized":"2024-04-25T14:03:21.445677+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449716+0000","last_change":"2024-04-25T14:03:21.449716+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449716+0000","last_undegraded":"2024-04-25T14:03:21.449716+0000","last_fullsized":"2024-04-25T14:03:21.449716+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451940+0000","last_change":"2024-04-25T14:03:21.451940+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451940+0000","last_undegraded":"2024-04-25T14:03:21.451940+0000","last_fullsized":"2024-04-25T14:03:21.451940+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449662+0000","last_change":"2024-04-25T14:03:21.449662+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449662+0000","last_undegraded":"2024-04-25T14:03:21.449662+0000","last_fullsized":"2024-04-25T14:03:21.449662+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449170+0000","last_change":"2024-04-25T14:03:21.449170+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449170+0000","last_undegraded":"2024-04-25T14:03:21.449170+0000","last_fullsized":"2024-04-25T14:03:21.449170+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448619+0000","last_change":"2024-04-25T14:03:21.448619+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448619+0000","last_undegraded":"2024-04-25T14:03:21.448619+0000","last_fullsized":"2024-04-25T14:03:21.448619+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449266+0000","last_change":"2024-04-25T14:03:21.449266+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449266+0000","last_undegraded":"2024-04-25T14:03:21.449266+0000","last_fullsized":"2024-04-25T14:03:21.449266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451353+0000","last_change":"2024-04-25T14:03:21.451353+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451353+0000","last_undegraded":"2024-04-25T14:03:21.451353+0000","last_fullsized":"2024-04-25T14:03:21.451353+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451912+0000","last_change":"2024-04-25T14:03:21.451912+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451912+0000","last_undegraded":"2024-04-25T14:03:21.451912+0000","last_fullsized":"2024-04-25T14:03:21.451912+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448729+0000","last_change":"2024-04-25T14:03:21.448729+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448729+0000","last_undegraded":"2024-04-25T14:03:21.448729+0000","last_fullsized":"2024-04-25T14:03:21.448729+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445331+0000","last_change":"2024-04-25T14:03:21.445331+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445331+0000","last_undegraded":"2024-04-25T14:03:21.445331+0000","last_fullsized":"2024-04-25T14:03:21.445331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449046+0000","last_change":"2024-04-25T14:03:21.449046+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449046+0000","last_undegraded":"2024-04-25T14:03:21.449046+0000","last_fullsized":"2024-04-25T14:03:21.449046+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444727+0000","last_change":"2024-04-25T14:03:21.444727+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444727+0000","last_undegraded":"2024-04-25T14:03:21.444727+0000","last_fullsized":"2024-04-25T14:03:21.444727+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":140,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-25T14:03:21.445636+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:03:21.445636+0000","last_peered":"2024-04-25T14:03:21.445636+0000","last_clean":"2024-04-25T14:03:21.445636+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:03:21.445636+0000","last_undegraded":"2024-04-25T14:03:21.445636+0000","last_fullsized":"2024-04-25T14:03:21.445636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":759878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:03:27.837 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:03:27.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.982+0000 7fb0bd39d640 1 Processor -- start 2024-04-25T14:03:27.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.982+0000 7fb0bd39d640 1 -- start start 2024-04-25T14:03:27.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb0b80d3d30 0x7fb0b80b4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b8160690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb0b8160bd0 0x7fb0b8163c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b80d4140 con 0x7fb0b8160bd0 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb0b806c7a0 con 0x7fb0b80d3d30 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bd39d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b80b51c0 con 0x7fb0b80d4520 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb0b8160bd0 0x7fb0b8163c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb0b8160bd0 0x7fb0b8163c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45144/0 (socket says 172.21.15.150:45144) 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 -- 172.21.15.150:0/49316770 learned_addr learned my addr 172.21.15.150:0/49316770 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0b77fe640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b8160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0b7fff640 1 --2- 172.21.15.150:0/49316770 >> v2:172.21.15.187:3300/0 conn(0x7fb0b80d3d30 0x7fb0b80b4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 -- 172.21.15.150:0/49316770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 msgr2=0x7fb0b8160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b8160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 -- 172.21.15.150:0/49316770 >> v2:172.21.15.187:3300/0 conn(0x7fb0b80d3d30 msgr2=0x7fb0b80b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:27.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 --2- 172.21.15.150:0/49316770 >> v2:172.21.15.187:3300/0 conn(0x7fb0b80d3d30 0x7fb0b80b4c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.985 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.984+0000 7fb0bcb9c640 1 -- 172.21.15.150:0/49316770 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b80d4d00 con 0x7fb0b8160bd0 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.985+0000 7fb0bcb9c640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8163c40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb0b808e670 tx=0x7fb0b002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cca5616fcb6af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.985+0000 7fb0b6ffd640 1 -- 172.21.15.150:0/49316770 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0036070 con 0x7fb0b8160bd0 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.985+0000 7fb0b6ffd640 1 -- 172.21.15.150:0/49316770 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0b0031600 con 0x7fb0b8160bd0 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.985+0000 7fb0b6ffd640 1 -- 172.21.15.150:0/49316770 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00319d0 con 0x7fb0b8160bd0 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 -- 172.21.15.150:0/49316770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b8160bd0 msgr2=0x7fb0b8163c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8163c40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb0b808e670 tx=0x7fb0b002a6a0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 -- 172.21.15.150:0/49316770 shutdown_connections 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8163c40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/49316770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b8160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/49316770 >> v2:172.21.15.187:3300/0 conn(0x7fb0b80d3d30 0x7fb0b80b4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 -- 172.21.15.150:0/49316770 >> 172.21.15.150:0/49316770 conn(0x7fb0b80b7440 msgr2=0x7fb0b80b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 -- 172.21.15.150:0/49316770 shutdown_connections 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.986+0000 7fb0bd39d640 1 -- 172.21.15.150:0/49316770 wait complete. 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.988+0000 7fb0bd39d640 1 Processor -- start 2024-04-25T14:03:27.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.988+0000 7fb0bd39d640 1 -- start start 2024-04-25T14:03:27.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b815fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8160fe0 con 0x7fb0b80d3d30 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b80d52c0 con 0x7fb0b8160bd0 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bd39d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb0b80d4930 con 0x7fb0b80d4520 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0b7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45146/0 (socket says 172.21.15.150:45146) 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0b7fff640 1 -- 172.21.15.150:0/1669921061 learned_addr learned my addr 172.21.15.150:0/1669921061 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0b77fe640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b815fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.989+0000 7fb0bcb9c640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 -- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 msgr2=0x7fb0b815fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b815fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 -- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0b8160bd0 msgr2=0x7fb0b8160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:27.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 -- 172.21.15.150:0/1669921061 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b81fa230 con 0x7fb0b80d3d30 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b7fff640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800d620 tx=0x7fb0a800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800b620 con 0x7fb0b80d3d30 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0a8019070 con 0x7fb0b80d3d30 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a800ba00 con 0x7fb0b80d3d30 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b81fa510 con 0x7fb0b80d3d30 2024-04-25T14:03:27.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.990+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b8155720 con 0x7fb0b80d3d30 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.991+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b8050740 con 0x7fb0b80d3d30 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.992+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb0a800bba0 con 0x7fb0b80d3d30 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.992+0000 7fb0b4ff9640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 0x7fb0a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.992+0000 7fb0b77fe640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 0x7fb0a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.992+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fb0a80956a0 con 0x7fb0b80d3d30 2024-04-25T14:03:27.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.993+0000 7fb0b77fe640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 0x7fb0a40769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80cc650 tx=0x7fb0ac008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:27.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:27.997+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb0a805f650 con 0x7fb0b80d3d30 2024-04-25T14:03:28.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.167+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb0b815f3a0 con 0x7fb0a40744b0 2024-04-25T14:03:28.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.173+0000 7fb0b4ff9640 1 -- 172.21.15.150:0/1669921061 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240060 (secure 0 0 0) 0x7fb0b815f3a0 con 0x7fb0a40744b0 2024-04-25T14:03:28.174 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:28.176 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:03:28.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.175+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 msgr2=0x7fb0a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.175+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 0x7fb0a40769c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb0b80cc650 tx=0x7fb0ac008040 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.175+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 msgr2=0x7fb0b81f9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.175+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800d620 tx=0x7fb0a800dae0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.175+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 shutdown_connections 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb0a40744b0 0x7fb0a40769c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb0b8160bd0 0x7fb0b8160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb0b80d4520 0x7fb0b815fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 --2- 172.21.15.150:0/1669921061 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb0b80d3d30 0x7fb0b81f9a00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 >> 172.21.15.150:0/1669921061 conn(0x7fb0b80b7440 msgr2=0x7fb0b80b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 shutdown_connections 2024-04-25T14:03:28.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.176+0000 7fb0bd39d640 1 -- 172.21.15.150:0/1669921061 wait complete. 2024-04-25T14:03:28.186 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":76,"stamp":"2024-04-25T14:03:26.603503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220416,"kb_used_data":4212,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754304,"statfs":{"total":773094113280,"available":772868407296,"internally_reserved":0,"allocated":4313088,"data_stored":2493353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.160575"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448728+0000","last_change":"2024-04-25T14:03:21.448728+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448728+0000","last_undegraded":"2024-04-25T14:03:21.448728+0000","last_fullsized":"2024-04-25T14:03:21.448728+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451883+0000","last_change":"2024-04-25T14:03:21.451883+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451883+0000","last_undegraded":"2024-04-25T14:03:21.451883+0000","last_fullsized":"2024-04-25T14:03:21.451883+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451477+0000","last_change":"2024-04-25T14:03:21.451477+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451477+0000","last_undegraded":"2024-04-25T14:03:21.451477+0000","last_fullsized":"2024-04-25T14:03:21.451477+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449012+0000","last_change":"2024-04-25T14:03:21.449012+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449012+0000","last_undegraded":"2024-04-25T14:03:21.449012+0000","last_fullsized":"2024-04-25T14:03:21.449012+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444412+0000","last_change":"2024-04-25T14:03:21.444412+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444412+0000","last_undegraded":"2024-04-25T14:03:21.444412+0000","last_fullsized":"2024-04-25T14:03:21.444412+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451346+0000","last_change":"2024-04-25T14:03:21.451346+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451346+0000","last_undegraded":"2024-04-25T14:03:21.451346+0000","last_fullsized":"2024-04-25T14:03:21.451346+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445316+0000","last_change":"2024-04-25T14:03:21.445316+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445316+0000","last_undegraded":"2024-04-25T14:03:21.445316+0000","last_fullsized":"2024-04-25T14:03:21.445316+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448560+0000","last_change":"2024-04-25T14:03:21.448560+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448560+0000","last_undegraded":"2024-04-25T14:03:21.448560+0000","last_fullsized":"2024-04-25T14:03:21.448560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449111+0000","last_change":"2024-04-25T14:03:21.449111+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449111+0000","last_undegraded":"2024-04-25T14:03:21.449111+0000","last_fullsized":"2024-04-25T14:03:21.449111+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448969+0000","last_change":"2024-04-25T14:03:21.448969+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448969+0000","last_undegraded":"2024-04-25T14:03:21.448969+0000","last_fullsized":"2024-04-25T14:03:21.448969+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449004+0000","last_change":"2024-04-25T14:03:21.449004+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449004+0000","last_undegraded":"2024-04-25T14:03:21.449004+0000","last_fullsized":"2024-04-25T14:03:21.449004+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451914+0000","last_change":"2024-04-25T14:03:21.451914+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451914+0000","last_undegraded":"2024-04-25T14:03:21.451914+0000","last_fullsized":"2024-04-25T14:03:21.451914+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448722+0000","last_change":"2024-04-25T14:03:21.448722+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448722+0000","last_undegraded":"2024-04-25T14:03:21.448722+0000","last_fullsized":"2024-04-25T14:03:21.448722+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444308+0000","last_change":"2024-04-25T14:03:21.444308+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444308+0000","last_undegraded":"2024-04-25T14:03:21.444308+0000","last_fullsized":"2024-04-25T14:03:21.444308+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449771+0000","last_change":"2024-04-25T14:03:21.449771+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449771+0000","last_undegraded":"2024-04-25T14:03:21.449771+0000","last_fullsized":"2024-04-25T14:03:21.449771+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445677+0000","last_change":"2024-04-25T14:03:21.445677+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445677+0000","last_undegraded":"2024-04-25T14:03:21.445677+0000","last_fullsized":"2024-04-25T14:03:21.445677+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449716+0000","last_change":"2024-04-25T14:03:21.449716+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449716+0000","last_undegraded":"2024-04-25T14:03:21.449716+0000","last_fullsized":"2024-04-25T14:03:21.449716+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451940+0000","last_change":"2024-04-25T14:03:21.451940+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451940+0000","last_undegraded":"2024-04-25T14:03:21.451940+0000","last_fullsized":"2024-04-25T14:03:21.451940+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449662+0000","last_change":"2024-04-25T14:03:21.449662+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449662+0000","last_undegraded":"2024-04-25T14:03:21.449662+0000","last_fullsized":"2024-04-25T14:03:21.449662+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449170+0000","last_change":"2024-04-25T14:03:21.449170+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449170+0000","last_undegraded":"2024-04-25T14:03:21.449170+0000","last_fullsized":"2024-04-25T14:03:21.449170+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444291+0000","last_change":"2024-04-25T14:03:21.444291+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444291+0000","last_undegraded":"2024-04-25T14:03:21.444291+0000","last_fullsized":"2024-04-25T14:03:21.444291+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448619+0000","last_change":"2024-04-25T14:03:21.448619+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448619+0000","last_undegraded":"2024-04-25T14:03:21.448619+0000","last_fullsized":"2024-04-25T14:03:21.448619+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449266+0000","last_change":"2024-04-25T14:03:21.449266+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449266+0000","last_undegraded":"2024-04-25T14:03:21.449266+0000","last_fullsized":"2024-04-25T14:03:21.449266+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448553+0000","last_change":"2024-04-25T14:03:21.448553+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448553+0000","last_undegraded":"2024-04-25T14:03:21.448553+0000","last_fullsized":"2024-04-25T14:03:21.448553+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451353+0000","last_change":"2024-04-25T14:03:21.451353+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451353+0000","last_undegraded":"2024-04-25T14:03:21.451353+0000","last_fullsized":"2024-04-25T14:03:21.451353+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.451912+0000","last_change":"2024-04-25T14:03:21.451912+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.451912+0000","last_undegraded":"2024-04-25T14:03:21.451912+0000","last_fullsized":"2024-04-25T14:03:21.451912+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.448729+0000","last_change":"2024-04-25T14:03:21.448729+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.448729+0000","last_undegraded":"2024-04-25T14:03:21.448729+0000","last_fullsized":"2024-04-25T14:03:21.448729+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.445331+0000","last_change":"2024-04-25T14:03:21.445331+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.445331+0000","last_undegraded":"2024-04-25T14:03:21.445331+0000","last_fullsized":"2024-04-25T14:03:21.445331+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.449046+0000","last_change":"2024-04-25T14:03:21.449046+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.449046+0000","last_undegraded":"2024-04-25T14:03:21.449046+0000","last_fullsized":"2024-04-25T14:03:21.449046+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-25T14:03:22.440461+0000","last_change":"2024-04-25T14:03:22.440461+0000","last_active":"2024-04-25T14:03:22.440461+0000","last_peered":"2024-04-25T14:03:22.440461+0000","last_clean":"2024-04-25T14:03:22.440461+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:22.440461+0000","last_undegraded":"2024-04-25T14:03:22.440461+0000","last_fullsized":"2024-04-25T14:03:22.440461+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":35,"state":"creating+peering","last_fresh":"2024-04-25T14:03:21.444727+0000","last_change":"2024-04-25T14:03:21.444727+0000","last_active":"2024-04-25T14:03:21.433596+0000","last_peered":"2024-04-25T14:03:21.433596+0000","last_clean":"2024-04-25T14:03:21.433596+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-25T14:03:21.444727+0000","last_undegraded":"2024-04-25T14:03:21.444727+0000","last_fullsized":"2024-04-25T14:03:21.444727+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":140,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-25T14:03:21.445636+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:03:21.445636+0000","last_peered":"2024-04-25T14:03:21.445636+0000","last_clean":"2024-04-25T14:03:21.445636+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:03:21.445636+0000","last_undegraded":"2024-04-25T14:03:21.445636+0000","last_fullsized":"2024-04-25T14:03:21.445636+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":983040,"data_stored":753669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27424,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":163301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":759878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.190 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-25T14:03:28.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-25T14:03:28.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.191 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-25T14:03:28.191 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-25T14:03:28.192 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-25T14:03:28.193 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-04-25T14:03:28.194 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-04-25T14:03:28.195 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-25T14:03:28.196 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448728+0000', 'last_change': '2024-04-25T14:03:21.448728+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448728+0000', 'last_undegraded': '2024-04-25T14:03:21.448728+0000', 'last_fullsized': '2024-04-25T14:03:21.448728+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451883+0000', 'last_change': '2024-04-25T14:03:21.451883+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451883+0000', 'last_undegraded': '2024-04-25T14:03:21.451883+0000', 'last_fullsized': '2024-04-25T14:03:21.451883+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451477+0000', 'last_change': '2024-04-25T14:03:21.451477+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451477+0000', 'last_undegraded': '2024-04-25T14:03:21.451477+0000', 'last_fullsized': '2024-04-25T14:03:21.451477+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T14:03:28.197 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.444291+0000', 'last_change': '2024-04-25T14:03:21.444291+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.444291+0000', 'last_undegraded': '2024-04-25T14:03:21.444291+0000', 'last_fullsized': '2024-04-25T14:03:21.444291+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449012+0000', 'last_change': '2024-04-25T14:03:21.449012+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449012+0000', 'last_undegraded': '2024-04-25T14:03:21.449012+0000', 'last_fullsized': '2024-04-25T14:03:21.449012+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.198 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.444412+0000', 'last_change': '2024-04-25T14:03:21.444412+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.444412+0000', 'last_undegraded': '2024-04-25T14:03:21.444412+0000', 'last_fullsized': '2024-04-25T14:03:21.444412+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451346+0000', 'last_change': '2024-04-25T14:03:21.451346+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451346+0000', 'last_undegraded': '2024-04-25T14:03:21.451346+0000', 'last_fullsized': '2024-04-25T14:03:21.451346+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.445316+0000', 'last_change': '2024-04-25T14:03:21.445316+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.445316+0000', 'last_undegraded': '2024-04-25T14:03:21.445316+0000', 'last_fullsized': '2024-04-25T14:03:21.445316+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-25T14:03:28.199 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448560+0000', 'last_change': '2024-04-25T14:03:21.448560+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448560+0000', 'last_undegraded': '2024-04-25T14:03:21.448560+0000', 'last_fullsized': '2024-04-25T14:03:21.448560+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449111+0000', 'last_change': '2024-04-25T14:03:21.449111+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449111+0000', 'last_undegraded': '2024-04-25T14:03:21.449111+0000', 'last_fullsized': '2024-04-25T14:03:21.449111+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448969+0000', 'last_change': '2024-04-25T14:03:21.448969+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448969+0000', 'last_undegraded': '2024-04-25T14:03:21.448969+0000', 'last_fullsized': '2024-04-25T14:03:21.448969+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-04-25T14:03:28.200 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448553+0000', 'last_change': '2024-04-25T14:03:21.448553+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448553+0000', 'last_undegraded': '2024-04-25T14:03:21.448553+0000', 'last_fullsized': '2024-04-25T14:03:21.448553+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-04-25T14:03:28.201 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449004+0000', 'last_change': '2024-04-25T14:03:21.449004+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449004+0000', 'last_undegraded': '2024-04-25T14:03:21.449004+0000', 'last_fullsized': '2024-04-25T14:03:21.449004+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451914+0000', 'last_change': '2024-04-25T14:03:21.451914+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451914+0000', 'last_undegraded': '2024-04-25T14:03:21.451914+0000', 'last_fullsized': '2024-04-25T14:03:21.451914+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448722+0000', 'last_change': '2024-04-25T14:03:21.448722+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448722+0000', 'last_undegraded': '2024-04-25T14:03:21.448722+0000', 'last_fullsized': '2024-04-25T14:03:21.448722+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-25T14:03:28.202 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.444308+0000', 'last_change': '2024-04-25T14:03:21.444308+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.444308+0000', 'last_undegraded': '2024-04-25T14:03:21.444308+0000', 'last_fullsized': '2024-04-25T14:03:21.444308+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449771+0000', 'last_change': '2024-04-25T14:03:21.449771+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449771+0000', 'last_undegraded': '2024-04-25T14:03:21.449771+0000', 'last_fullsized': '2024-04-25T14:03:21.449771+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.445677+0000', 'last_change': '2024-04-25T14:03:21.445677+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.445677+0000', 'last_undegraded': '2024-04-25T14:03:21.445677+0000', 'last_fullsized': '2024-04-25T14:03:21.445677+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T14:03:28.203 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449716+0000', 'last_change': '2024-04-25T14:03:21.449716+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449716+0000', 'last_undegraded': '2024-04-25T14:03:21.449716+0000', 'last_fullsized': '2024-04-25T14:03:21.449716+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451940+0000', 'last_change': '2024-04-25T14:03:21.451940+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451940+0000', 'last_undegraded': '2024-04-25T14:03:21.451940+0000', 'last_fullsized': '2024-04-25T14:03:21.451940+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-04-25T14:03:28.204 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449662+0000', 'last_change': '2024-04-25T14:03:21.449662+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449662+0000', 'last_undegraded': '2024-04-25T14:03:21.449662+0000', 'last_fullsized': '2024-04-25T14:03:21.449662+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449170+0000', 'last_change': '2024-04-25T14:03:21.449170+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449170+0000', 'last_undegraded': '2024-04-25T14:03:21.449170+0000', 'last_fullsized': '2024-04-25T14:03:21.449170+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.444291+0000', 'last_change': '2024-04-25T14:03:21.444291+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.444291+0000', 'last_undegraded': '2024-04-25T14:03:21.444291+0000', 'last_fullsized': '2024-04-25T14:03:21.444291+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-04-25T14:03:28.205 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448619+0000', 'last_change': '2024-04-25T14:03:21.448619+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448619+0000', 'last_undegraded': '2024-04-25T14:03:21.448619+0000', 'last_fullsized': '2024-04-25T14:03:21.448619+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449266+0000', 'last_change': '2024-04-25T14:03:21.449266+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449266+0000', 'last_undegraded': '2024-04-25T14:03:21.449266+0000', 'last_fullsized': '2024-04-25T14:03:21.449266+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448553+0000', 'last_change': '2024-04-25T14:03:21.448553+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448553+0000', 'last_undegraded': '2024-04-25T14:03:21.448553+0000', 'last_fullsized': '2024-04-25T14:03:21.448553+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-25T14:03:28.206 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451353+0000', 'last_change': '2024-04-25T14:03:21.451353+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451353+0000', 'last_undegraded': '2024-04-25T14:03:21.451353+0000', 'last_fullsized': '2024-04-25T14:03:21.451353+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.451912+0000', 'last_change': '2024-04-25T14:03:21.451912+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.451912+0000', 'last_undegraded': '2024-04-25T14:03:21.451912+0000', 'last_fullsized': '2024-04-25T14:03:21.451912+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.448729+0000', 'last_change': '2024-04-25T14:03:21.448729+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.448729+0000', 'last_undegraded': '2024-04-25T14:03:21.448729+0000', 'last_fullsized': '2024-04-25T14:03:21.448729+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-04-25T14:03:28.207 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.445331+0000', 'last_change': '2024-04-25T14:03:21.445331+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.445331+0000', 'last_undegraded': '2024-04-25T14:03:21.445331+0000', 'last_fullsized': '2024-04-25T14:03:21.445331+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.449046+0000', 'last_change': '2024-04-25T14:03:21.449046+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.449046+0000', 'last_undegraded': '2024-04-25T14:03:21.449046+0000', 'last_fullsized': '2024-04-25T14:03:21.449046+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-25T14:03:22.440461+0000', 'last_change': '2024-04-25T14:03:22.440461+0000', 'last_active': '2024-04-25T14:03:22.440461+0000', 'last_peered': '2024-04-25T14:03:22.440461+0000', 'last_clean': '2024-04-25T14:03:22.440461+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:22.440461+0000', 'last_undegraded': '2024-04-25T14:03:22.440461+0000', 'last_fullsized': '2024-04-25T14:03:22.440461+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:22.440461+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-04-25T14:03:28.208 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 35, 'state': 'creating+peering', 'last_fresh': '2024-04-25T14:03:21.444727+0000', 'last_change': '2024-04-25T14:03:21.444727+0000', 'last_active': '2024-04-25T14:03:21.433596+0000', 'last_peered': '2024-04-25T14:03:21.433596+0000', 'last_clean': '2024-04-25T14:03:21.433596+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-25T14:03:21.444727+0000', 'last_undegraded': '2024-04-25T14:03:21.444727+0000', 'last_fullsized': '2024-04-25T14:03:21.444727+0000', 'mapping_epoch': 35, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 35, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'last_clean_scrub_stamp': '2024-04-25T14:03:21.433596+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-25T14:03:28.209 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-25T14:03:28.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.336+0000 7f6834514640 1 Processor -- start 2024-04-25T14:03:28.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.336+0000 7f6834514640 1 -- start start 2024-04-25T14:03:28.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f6834514640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c16b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f6834514640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c06c570 con 0x7f682c1601d0 2024-04-25T14:03:28.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f6834514640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682c1605e0 con 0x7f682c0b9070 2024-04-25T14:03:28.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f6834514640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f682c0d5980 con 0x7f682c06da00 2024-04-25T14:03:28.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f68322b0640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f682c1601d0 0x7f682c0d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34478/0 (socket says 172.21.15.150:34478) 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f68322b0640 1 -- 172.21.15.150:0/2558130876 learned_addr learned my addr 172.21.15.150:0/2558130876 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.337+0000 7f6832ab1640 1 --2- 172.21.15.150:0/2558130876 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c16b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468479022 0 0) 0x7f682c06c570 con 0x7f682c1601d0 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6818003520 con 0x7f682c1601d0 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2015284173 0 0) 0x7f682c1605e0 con 0x7f682c0b9070 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f682c06c570 con 0x7f682c0b9070 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f6832ab1640 1 -- 172.21.15.150:0/2558130876 >> v1:172.21.15.187:6789/0 conn(0x7f682c0b9070 legacy=0x7f682c06d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f6832ab1640 1 -- 172.21.15.150:0/2558130876 >> v1:172.21.15.150:6789/0 conn(0x7f682c1601d0 legacy=0x7f682c0d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.338+0000 7f6832ab1640 1 -- 172.21.15.150:0/2558130876 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c06c810 con 0x7f682c06da00 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f6832ab1640 1 --2- 172.21.15.150:0/2558130876 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c16b070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f68240097d0 tx=0x7f682402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c00b305a3cfd5d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824037070 con 0x7f682c06da00 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6824032440 con 0x7f682c06da00 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f68312ae640 1 -- 172.21.15.150:0/2558130876 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6824032810 con 0x7f682c06da00 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f6834514640 1 -- 172.21.15.150:0/2558130876 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 msgr2=0x7f682c16b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.339+0000 7f6834514640 1 --2- 172.21.15.150:0/2558130876 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c16b070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f68240097d0 tx=0x7f682402a180 comp rx=0 tx=0).stop 2024-04-25T14:03:28.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.340+0000 7f6834514640 1 -- 172.21.15.150:0/2558130876 shutdown_connections 2024-04-25T14:03:28.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.340+0000 7f6834514640 1 --2- 172.21.15.150:0/2558130876 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c16b070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.340+0000 7f6834514640 1 -- 172.21.15.150:0/2558130876 >> 172.21.15.150:0/2558130876 conn(0x7f682c0d5d90 msgr2=0x7f682c0d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:28.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.340+0000 7f6834514640 1 -- 172.21.15.150:0/2558130876 shutdown_connections 2024-04-25T14:03:28.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.340+0000 7f6834514640 1 -- 172.21.15.150:0/2558130876 wait complete. 2024-04-25T14:03:28.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.341+0000 7f6834514640 1 Processor -- start 2024-04-25T14:03:28.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.341+0000 7f6834514640 1 -- start start 2024-04-25T14:03:28.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.342+0000 7f6834514640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 0x7f682c0cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f682c06de10 con 0x7f682c0b9070 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f682c06cdd0 con 0x7f682c1601d0 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f682c0b9480 con 0x7f682c06da00 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45150/0 (socket says 172.21.15.150:45150) 2024-04-25T14:03:28.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 -- 172.21.15.150:0/1047553991 learned_addr learned my addr 172.21.15.150:0/1047553991 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6832ab1640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 0x7f682c0cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f68322b0640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 -- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 msgr2=0x7f682c161980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c161980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 -- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 msgr2=0x7f682c0cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 0x7f682c0cd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 -- 172.21.15.150:0/1047553991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f682c1621d0 con 0x7f682c0b9070 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6832ab1640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 0x7f682c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6831aaf640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c3c0 tx=0x7f681c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c017070 con 0x7f682c0b9070 2024-04-25T14:03:28.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f681c0052f0 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681c003830 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f682c1624b0 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.343+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f682c162a70 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.344+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f682c161f40 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.345+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f681c01a070 con 0x7f682c0b9070 2024-04-25T14:03:28.347 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.346+0000 7f6822ffd640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 0x7f68100769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.346+0000 7f68322b0640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 0x7f68100769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.346+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f681c094600 con 0x7f682c0b9070 2024-04-25T14:03:28.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.346+0000 7f68322b0640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 0x7f68100769c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6828005c00 tx=0x7f6828009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.350+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f681c05e640 con 0x7f682c0b9070 2024-04-25T14:03:28.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.596+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f682c076a40 con 0x7f682c0b9070 2024-04-25T14:03:28.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.596+0000 7f6822ffd640 1 -- 172.21.15.150:0/1047553991 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1109 (secure 0 0 0) 0x7f681c062940 con 0x7f682c0b9070 2024-04-25T14:03:28.595 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:28.595 INFO:teuthology.orchestra.run.smithi150.stdout:{"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":38,"num_osds":8,"num_up_osds":8,"osd_up_since":1714053715,"num_in_osds":8,"osd_in_since":1714053706,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225705984,"bytes_avail":772868407296,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":16,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"c","status":"up:active","gid":5349}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":6,"modified":"2024-04-25T14:03:12.601334+0000","services":{}},"progress_events":{}} 2024-04-25T14:03:28.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 msgr2=0x7f68100769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 0x7f68100769c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6828005c00 tx=0x7f6828009040 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 msgr2=0x7f682c0ccdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f681c00c3c0 tx=0x7f681c00c880 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 shutdown_connections 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f68100744b0 0x7f68100769c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f682c1601d0 0x7f682c0cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f682c0b9070 0x7f682c0ccdf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 --2- 172.21.15.150:0/1047553991 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f682c06da00 0x7f682c161980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 >> 172.21.15.150:0/1047553991 conn(0x7f682c0d5d90 msgr2=0x7f682c0d03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 shutdown_connections 2024-04-25T14:03:28.598 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.598+0000 7f6834514640 1 -- 172.21.15.150:0/1047553991 wait complete. 2024-04-25T14:03:28.609 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-25T14:03:28.609 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:03:28.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.717+0000 7fd0548aa640 1 Processor -- start 2024-04-25T14:03:28.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.717+0000 7fd0548aa640 1 -- start start 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd0548aa640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd0500b5ab0 0x7fd0500cfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd0548aa640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0500b7f40 con 0x7fd050163690 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd0548aa640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd050164620 con 0x7fd0500b8700 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd0548aa640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd050164bc0 con 0x7fd0500b5ab0 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04e59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd050163690 0x7fd0500b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34490/0 (socket says 172.21.15.150:34490) 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04e59c640 1 -- 172.21.15.150:0/3696550945 learned_addr learned my addr 172.21.15.150:0/3696550945 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/3696550945 >> v2:172.21.15.187:3300/0 conn(0x7fd0500b5ab0 0x7fd0500cfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2978225155 0 0) 0x7fd0500b7f40 con 0x7fd050163690 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd038003520 con 0x7fd050163690 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.719+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1766127703 0 0) 0x7fd038003520 con 0x7fd050163690 2024-04-25T14:03:28.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 >> v2:172.21.15.187:3300/0 conn(0x7fd0500b5ab0 msgr2=0x7fd0500cfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 --2- 172.21.15.150:0/3696550945 >> v2:172.21.15.187:3300/0 conn(0x7fd0500b5ab0 0x7fd0500cfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 >> v1:172.21.15.187:6789/0 conn(0x7fd0500b8700 legacy=0x7fd0500b8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0500b3080 con 0x7fd050163690 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fd040001a10 con 0x7fd050163690 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 >> v1:172.21.15.150:6789/0 conn(0x7fd050163690 legacy=0x7fd0500b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0380039b0 0x7fd038003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd038004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd038005250 0x7fd03800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd050164bc0 con 0x7fd038004800 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd03800c1e0 con 0x7fd0380039b0 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd03800c3e0 con 0x7fd038005250 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd038004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04e59c640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0380039b0 0x7fd038003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04ed9d640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd038005250 0x7fd03800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 -- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd038005250 msgr2=0x7fd03800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd038005250 0x7fd03800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 -- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0380039b0 msgr2=0x7fd038003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0380039b0 0x7fd038003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04dd9b640 1 -- 172.21.15.150:0/3696550945 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd038003520 con 0x7fd038004800 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04ed9d640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd038005250 0x7fd03800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.720+0000 7fd04ed9d640 1 -- 172.21.15.150:0/3696550945 reap_dead start 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.721+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd038004c10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd044009ac0 tx=0x7fd038005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67c90666de63ffd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.721+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044036070 con 0x7fd038004800 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.721+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd044031400 con 0x7fd038004800 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.721+0000 7fd04d59a640 1 -- 172.21.15.150:0/3696550945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd044009690 con 0x7fd038004800 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 -- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 msgr2=0x7fd038004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd038004c10 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fd044009ac0 tx=0x7fd038005830 comp rx=0 tx=0).stop 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 -- 172.21.15.150:0/3696550945 shutdown_connections 2024-04-25T14:03:28.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 --2- 172.21.15.150:0/3696550945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd038004c10 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 -- 172.21.15.150:0/3696550945 >> 172.21.15.150:0/3696550945 conn(0x7fd0500d03a0 msgr2=0x7fd0500d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:28.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 -- 172.21.15.150:0/3696550945 shutdown_connections 2024-04-25T14:03:28.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.722+0000 7fd0548aa640 1 -- 172.21.15.150:0/3696550945 wait complete. 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.724+0000 7fd0548aa640 1 Processor -- start 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.724+0000 7fd0548aa640 1 -- start start 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0500b4a60 0x7fd0501f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050148120 0x7fd050148530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500b2ca0 con 0x7fd038004800 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd0500b81e0 con 0x7fd0500b4a60 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd0548aa640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd0500b2e20 con 0x7fd050148120 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45166/0 (socket says 172.21.15.150:45166) 2024-04-25T14:03:28.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 -- 172.21.15.150:0/1015759159 learned_addr learned my addr 172.21.15.150:0/1015759159 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 -- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050148120 msgr2=0x7fd050148530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050148120 0x7fd050148530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 -- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0500b4a60 msgr2=0x7fd0501f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0500b4a60 0x7fd0501f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.725+0000 7fd04e59c640 1 -- 172.21.15.150:0/1015759159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0501e9fc0 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd04e59c640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd0400079d0 tx=0x7fd04000a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040017020 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd040007ac0 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd040013630 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0500b3f80 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.726+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0500b4540 con 0x7fd038004800 2024-04-25T14:03:28.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.727+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0500b3d70 con 0x7fd038004800 2024-04-25T14:03:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.727+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd04001a020 con 0x7fd038004800 2024-04-25T14:03:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.728+0000 7fd03effd640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 0x7fd0300769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.728+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd040095e80 con 0x7fd038004800 2024-04-25T14:03:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.728+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 0x7fd0300769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:28.728 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.728+0000 7fd04dd9b640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 0x7fd0300769c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd044033000 tx=0x7fd04402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:28.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.733+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd040021020 con 0x7fd038004800 2024-04-25T14:03:28.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.902+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd050076a40 con 0x7fd0300744b0 2024-04-25T14:03:28.906 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.908+0000 7fd03effd640 1 -- 172.21.15.150:0/1015759159 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7fd050076a40 con 0x7fd0300744b0 2024-04-25T14:03:28.909 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:28.911 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 msgr2=0x7fd0300769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 0x7fd0300769c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd044033000 tx=0x7fd04402e000 comp rx=0 tx=0).stop 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 msgr2=0x7fd05006e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd0400079d0 tx=0x7fd04000a730 comp rx=0 tx=0).stop 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 shutdown_connections 2024-04-25T14:03:28.912 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd0300744b0 0x7fd0300769c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd050148120 0x7fd050148530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd0500b4a60 0x7fd0501f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 --2- 172.21.15.150:0/1015759159 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd038004800 0x7fd05006e660 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 >> 172.21.15.150:0/1015759159 conn(0x7fd0500d03a0 msgr2=0x7fd05015f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 shutdown_connections 2024-04-25T14:03:28.913 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:28.911+0000 7fd0548aa640 1 -- 172.21.15.150:0/1015759159 wait complete. 2024-04-25T14:03:28.921 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-04-25T14:03:28.603911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220816,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753904,"statfs":{"total":773094113280,"available":772867997696,"internally_reserved":0,"allocated":4759552,"data_stored":2705844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.158352"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627620+0000","last_change":"2024-04-25T14:03:23.474092+0000","last_active":"2024-04-25T14:03:25.627620+0000","last_peered":"2024-04-25T14:03:25.627620+0000","last_clean":"2024-04-25T14:03:25.627620+0000","last_became_active":"2024-04-25T14:03:23.473817+0000","last_became_peered":"2024-04-25T14:03:23.473817+0000","last_unstale":"2024-04-25T14:03:25.627620+0000","last_undegraded":"2024-04-25T14:03:25.627620+0000","last_fullsized":"2024-04-25T14:03:25.627620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:59.615179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627424+0000","last_change":"2024-04-25T14:03:23.471063+0000","last_active":"2024-04-25T14:03:25.627424+0000","last_peered":"2024-04-25T14:03:25.627424+0000","last_clean":"2024-04-25T14:03:25.627424+0000","last_became_active":"2024-04-25T14:03:23.470801+0000","last_became_peered":"2024-04-25T14:03:23.470801+0000","last_unstale":"2024-04-25T14:03:25.627424+0000","last_undegraded":"2024-04-25T14:03:25.627424+0000","last_fullsized":"2024-04-25T14:03:25.627424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526798+0000","last_change":"2024-04-25T14:03:23.475260+0000","last_active":"2024-04-25T14:03:24.526798+0000","last_peered":"2024-04-25T14:03:24.526798+0000","last_clean":"2024-04-25T14:03:24.526798+0000","last_became_active":"2024-04-25T14:03:23.474934+0000","last_became_peered":"2024-04-25T14:03:23.474934+0000","last_unstale":"2024-04-25T14:03:24.526798+0000","last_undegraded":"2024-04-25T14:03:24.526798+0000","last_fullsized":"2024-04-25T14:03:24.526798+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:19.446529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473756+0000","last_change":"2024-04-25T14:03:23.481483+0000","last_active":"2024-04-25T14:03:24.473756+0000","last_peered":"2024-04-25T14:03:24.473756+0000","last_clean":"2024-04-25T14:03:24.473756+0000","last_became_active":"2024-04-25T14:03:23.481184+0000","last_became_peered":"2024-04-25T14:03:23.481184+0000","last_unstale":"2024-04-25T14:03:24.473756+0000","last_undegraded":"2024-04-25T14:03:24.473756+0000","last_fullsized":"2024-04-25T14:03:24.473756+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:41.521048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473581+0000","last_change":"2024-04-25T14:03:23.480208+0000","last_active":"2024-04-25T14:03:24.473581+0000","last_peered":"2024-04-25T14:03:24.473581+0000","last_clean":"2024-04-25T14:03:24.473581+0000","last_became_active":"2024-04-25T14:03:23.479895+0000","last_became_peered":"2024-04-25T14:03:23.479895+0000","last_unstale":"2024-04-25T14:03:24.473581+0000","last_undegraded":"2024-04-25T14:03:24.473581+0000","last_fullsized":"2024-04-25T14:03:24.473581+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:12.824544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627237+0000","last_change":"2024-04-25T14:03:23.467851+0000","last_active":"2024-04-25T14:03:25.627237+0000","last_peered":"2024-04-25T14:03:25.627237+0000","last_clean":"2024-04-25T14:03:25.627237+0000","last_became_active":"2024-04-25T14:03:23.467550+0000","last_became_peered":"2024-04-25T14:03:23.467550+0000","last_unstale":"2024-04-25T14:03:25.627237+0000","last_undegraded":"2024-04-25T14:03:25.627237+0000","last_fullsized":"2024-04-25T14:03:25.627237+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:12.190368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473427+0000","last_change":"2024-04-25T14:03:23.467568+0000","last_active":"2024-04-25T14:03:24.473427+0000","last_peered":"2024-04-25T14:03:24.473427+0000","last_clean":"2024-04-25T14:03:24.473427+0000","last_became_active":"2024-04-25T14:03:23.467221+0000","last_became_peered":"2024-04-25T14:03:23.467221+0000","last_unstale":"2024-04-25T14:03:24.473427+0000","last_undegraded":"2024-04-25T14:03:24.473427+0000","last_fullsized":"2024-04-25T14:03:24.473427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:17.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627126+0000","last_change":"2024-04-25T14:03:23.481082+0000","last_active":"2024-04-25T14:03:25.627126+0000","last_peered":"2024-04-25T14:03:25.627126+0000","last_clean":"2024-04-25T14:03:25.627126+0000","last_became_active":"2024-04-25T14:03:23.480782+0000","last_became_peered":"2024-04-25T14:03:23.480782+0000","last_unstale":"2024-04-25T14:03:25.627126+0000","last_undegraded":"2024-04-25T14:03:25.627126+0000","last_fullsized":"2024-04-25T14:03:25.627126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526852+0000","last_change":"2024-04-25T14:03:23.479982+0000","last_active":"2024-04-25T14:03:24.526852+0000","last_peered":"2024-04-25T14:03:24.526852+0000","last_clean":"2024-04-25T14:03:24.526852+0000","last_became_active":"2024-04-25T14:03:23.479685+0000","last_became_peered":"2024-04-25T14:03:23.479685+0000","last_unstale":"2024-04-25T14:03:24.526852+0000","last_undegraded":"2024-04-25T14:03:24.526852+0000","last_fullsized":"2024-04-25T14:03:24.526852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:28.888914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471339+0000","last_change":"2024-04-25T14:03:23.476848+0000","last_active":"2024-04-25T14:03:24.471339+0000","last_peered":"2024-04-25T14:03:24.471339+0000","last_clean":"2024-04-25T14:03:24.471339+0000","last_became_active":"2024-04-25T14:03:23.476533+0000","last_became_peered":"2024-04-25T14:03:23.476533+0000","last_unstale":"2024-04-25T14:03:24.471339+0000","last_undegraded":"2024-04-25T14:03:24.471339+0000","last_fullsized":"2024-04-25T14:03:24.471339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472057+0000","last_change":"2024-04-25T14:03:23.481025+0000","last_active":"2024-04-25T14:03:24.472057+0000","last_peered":"2024-04-25T14:03:24.472057+0000","last_clean":"2024-04-25T14:03:24.472057+0000","last_became_active":"2024-04-25T14:03:23.480714+0000","last_became_peered":"2024-04-25T14:03:23.480714+0000","last_unstale":"2024-04-25T14:03:24.472057+0000","last_undegraded":"2024-04-25T14:03:24.472057+0000","last_fullsized":"2024-04-25T14:03:24.472057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526654+0000","last_change":"2024-04-25T14:03:23.473566+0000","last_active":"2024-04-25T14:03:24.526654+0000","last_peered":"2024-04-25T14:03:24.526654+0000","last_clean":"2024-04-25T14:03:24.526654+0000","last_became_active":"2024-04-25T14:03:23.473248+0000","last_became_peered":"2024-04-25T14:03:23.473248+0000","last_unstale":"2024-04-25T14:03:24.526654+0000","last_undegraded":"2024-04-25T14:03:24.526654+0000","last_fullsized":"2024-04-25T14:03:24.526654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:28.838095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473293+0000","last_change":"2024-04-25T14:03:23.469460+0000","last_active":"2024-04-25T14:03:24.473293+0000","last_peered":"2024-04-25T14:03:24.473293+0000","last_clean":"2024-04-25T14:03:24.473293+0000","last_became_active":"2024-04-25T14:03:23.469090+0000","last_became_peered":"2024-04-25T14:03:23.469090+0000","last_unstale":"2024-04-25T14:03:24.473293+0000","last_undegraded":"2024-04-25T14:03:24.473293+0000","last_fullsized":"2024-04-25T14:03:24.473293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:06:39.710834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471662+0000","last_change":"2024-04-25T14:03:23.478945+0000","last_active":"2024-04-25T14:03:24.471662+0000","last_peered":"2024-04-25T14:03:24.471662+0000","last_clean":"2024-04-25T14:03:24.471662+0000","last_became_active":"2024-04-25T14:03:23.478600+0000","last_became_peered":"2024-04-25T14:03:23.478600+0000","last_unstale":"2024-04-25T14:03:24.471662+0000","last_undegraded":"2024-04-25T14:03:24.471662+0000","last_fullsized":"2024-04-25T14:03:24.471662+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:48.357470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473313+0000","last_change":"2024-04-25T14:03:23.476367+0000","last_active":"2024-04-25T14:03:24.473313+0000","last_peered":"2024-04-25T14:03:24.473313+0000","last_clean":"2024-04-25T14:03:24.473313+0000","last_became_active":"2024-04-25T14:03:23.476047+0000","last_became_peered":"2024-04-25T14:03:23.476047+0000","last_unstale":"2024-04-25T14:03:24.473313+0000","last_undegraded":"2024-04-25T14:03:24.473313+0000","last_fullsized":"2024-04-25T14:03:24.473313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:59.615179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471103+0000","last_change":"2024-04-25T14:03:23.473411+0000","last_active":"2024-04-25T14:03:24.471103+0000","last_peered":"2024-04-25T14:03:24.471103+0000","last_clean":"2024-04-25T14:03:24.471103+0000","last_became_active":"2024-04-25T14:03:23.473087+0000","last_became_peered":"2024-04-25T14:03:23.473087+0000","last_unstale":"2024-04-25T14:03:24.471103+0000","last_undegraded":"2024-04-25T14:03:24.471103+0000","last_fullsized":"2024-04-25T14:03:24.471103+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:28.838095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473004+0000","last_change":"2024-04-25T14:03:23.479719+0000","last_active":"2024-04-25T14:03:24.473004+0000","last_peered":"2024-04-25T14:03:24.473004+0000","last_clean":"2024-04-25T14:03:24.473004+0000","last_became_active":"2024-04-25T14:03:23.479367+0000","last_became_peered":"2024-04-25T14:03:23.479367+0000","last_unstale":"2024-04-25T14:03:24.473004+0000","last_undegraded":"2024-04-25T14:03:24.473004+0000","last_fullsized":"2024-04-25T14:03:24.473004+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628123+0000","last_change":"2024-04-25T14:03:23.476703+0000","last_active":"2024-04-25T14:03:25.628123+0000","last_peered":"2024-04-25T14:03:25.628123+0000","last_clean":"2024-04-25T14:03:25.628123+0000","last_became_active":"2024-04-25T14:03:23.476391+0000","last_became_peered":"2024-04-25T14:03:23.476391+0000","last_unstale":"2024-04-25T14:03:25.628123+0000","last_undegraded":"2024-04-25T14:03:25.628123+0000","last_fullsized":"2024-04-25T14:03:25.628123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473932+0000","last_change":"2024-04-25T14:03:23.476020+0000","last_active":"2024-04-25T14:03:24.473932+0000","last_peered":"2024-04-25T14:03:24.473932+0000","last_clean":"2024-04-25T14:03:24.473932+0000","last_became_active":"2024-04-25T14:03:23.475456+0000","last_became_peered":"2024-04-25T14:03:23.475456+0000","last_unstale":"2024-04-25T14:03:24.473932+0000","last_undegraded":"2024-04-25T14:03:24.473932+0000","last_fullsized":"2024-04-25T14:03:24.473932+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:41.735868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627968+0000","last_change":"2024-04-25T14:03:23.480347+0000","last_active":"2024-04-25T14:03:25.627968+0000","last_peered":"2024-04-25T14:03:25.627968+0000","last_clean":"2024-04-25T14:03:25.627968+0000","last_became_active":"2024-04-25T14:03:23.480007+0000","last_became_peered":"2024-04-25T14:03:23.480007+0000","last_unstale":"2024-04-25T14:03:25.627968+0000","last_undegraded":"2024-04-25T14:03:25.627968+0000","last_fullsized":"2024-04-25T14:03:25.627968+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:48.357470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473453+0000","last_change":"2024-04-25T14:03:23.477606+0000","last_active":"2024-04-25T14:03:24.473453+0000","last_peered":"2024-04-25T14:03:24.473453+0000","last_clean":"2024-04-25T14:03:24.473453+0000","last_became_active":"2024-04-25T14:03:23.475489+0000","last_became_peered":"2024-04-25T14:03:23.475489+0000","last_unstale":"2024-04-25T14:03:24.473453+0000","last_undegraded":"2024-04-25T14:03:24.473453+0000","last_fullsized":"2024-04-25T14:03:24.473453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:46:25.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472926+0000","last_change":"2024-04-25T14:03:23.477678+0000","last_active":"2024-04-25T14:03:24.472926+0000","last_peered":"2024-04-25T14:03:24.472926+0000","last_clean":"2024-04-25T14:03:24.472926+0000","last_became_active":"2024-04-25T14:03:23.477360+0000","last_became_peered":"2024-04-25T14:03:23.477360+0000","last_unstale":"2024-04-25T14:03:24.472926+0000","last_undegraded":"2024-04-25T14:03:24.472926+0000","last_fullsized":"2024-04-25T14:03:24.472926+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:12.190368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472493+0000","last_change":"2024-04-25T14:03:23.475106+0000","last_active":"2024-04-25T14:03:24.472493+0000","last_peered":"2024-04-25T14:03:24.472493+0000","last_clean":"2024-04-25T14:03:24.472493+0000","last_became_active":"2024-04-25T14:03:23.474775+0000","last_became_peered":"2024-04-25T14:03:23.474775+0000","last_unstale":"2024-04-25T14:03:24.472493+0000","last_undegraded":"2024-04-25T14:03:24.472493+0000","last_fullsized":"2024-04-25T14:03:24.472493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473560+0000","last_change":"2024-04-25T14:03:23.472420+0000","last_active":"2024-04-25T14:03:24.473560+0000","last_peered":"2024-04-25T14:03:24.473560+0000","last_clean":"2024-04-25T14:03:24.473560+0000","last_became_active":"2024-04-25T14:03:23.472051+0000","last_became_peered":"2024-04-25T14:03:23.472051+0000","last_unstale":"2024-04-25T14:03:24.473560+0000","last_undegraded":"2024-04-25T14:03:24.473560+0000","last_fullsized":"2024-04-25T14:03:24.473560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:16.801664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628042+0000","last_change":"2024-04-25T14:03:23.477127+0000","last_active":"2024-04-25T14:03:25.628042+0000","last_peered":"2024-04-25T14:03:25.628042+0000","last_clean":"2024-04-25T14:03:25.628042+0000","last_became_active":"2024-04-25T14:03:23.473390+0000","last_became_peered":"2024-04-25T14:03:23.473390+0000","last_unstale":"2024-04-25T14:03:25.628042+0000","last_undegraded":"2024-04-25T14:03:25.628042+0000","last_fullsized":"2024-04-25T14:03:25.628042+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:44.718503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627397+0000","last_change":"2024-04-25T14:03:23.478890+0000","last_active":"2024-04-25T14:03:25.627397+0000","last_peered":"2024-04-25T14:03:25.627397+0000","last_clean":"2024-04-25T14:03:25.627397+0000","last_became_active":"2024-04-25T14:03:23.478468+0000","last_became_peered":"2024-04-25T14:03:23.478468+0000","last_unstale":"2024-04-25T14:03:25.627397+0000","last_undegraded":"2024-04-25T14:03:25.627397+0000","last_fullsized":"2024-04-25T14:03:25.627397+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:50.950138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628023+0000","last_change":"2024-04-25T14:03:23.472415+0000","last_active":"2024-04-25T14:03:25.628023+0000","last_peered":"2024-04-25T14:03:25.628023+0000","last_clean":"2024-04-25T14:03:25.628023+0000","last_became_active":"2024-04-25T14:03:23.472112+0000","last_became_peered":"2024-04-25T14:03:23.472112+0000","last_unstale":"2024-04-25T14:03:25.628023+0000","last_undegraded":"2024-04-25T14:03:25.628023+0000","last_fullsized":"2024-04-25T14:03:25.628023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:03.312923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471426+0000","last_change":"2024-04-25T14:03:23.473541+0000","last_active":"2024-04-25T14:03:24.471426+0000","last_peered":"2024-04-25T14:03:24.471426+0000","last_clean":"2024-04-25T14:03:24.471426+0000","last_became_active":"2024-04-25T14:03:23.473227+0000","last_became_peered":"2024-04-25T14:03:23.473227+0000","last_unstale":"2024-04-25T14:03:24.471426+0000","last_undegraded":"2024-04-25T14:03:24.471426+0000","last_fullsized":"2024-04-25T14:03:24.471426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627818+0000","last_change":"2024-04-25T14:03:23.468805+0000","last_active":"2024-04-25T14:03:25.627818+0000","last_peered":"2024-04-25T14:03:25.627818+0000","last_clean":"2024-04-25T14:03:25.627818+0000","last_became_active":"2024-04-25T14:03:23.468502+0000","last_became_peered":"2024-04-25T14:03:23.468502+0000","last_unstale":"2024-04-25T14:03:25.627818+0000","last_undegraded":"2024-04-25T14:03:25.627818+0000","last_fullsized":"2024-04-25T14:03:25.627818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:45:48.633548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627826+0000","last_change":"2024-04-25T14:03:23.469483+0000","last_active":"2024-04-25T14:03:25.627826+0000","last_peered":"2024-04-25T14:03:25.627826+0000","last_clean":"2024-04-25T14:03:25.627826+0000","last_became_active":"2024-04-25T14:03:23.469191+0000","last_became_peered":"2024-04-25T14:03:23.469191+0000","last_unstale":"2024-04-25T14:03:25.627826+0000","last_undegraded":"2024-04-25T14:03:25.627826+0000","last_fullsized":"2024-04-25T14:03:25.627826+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:40.771209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472686+0000","last_change":"2024-04-25T14:03:23.468203+0000","last_active":"2024-04-25T14:03:24.472686+0000","last_peered":"2024-04-25T14:03:24.472686+0000","last_clean":"2024-04-25T14:03:24.472686+0000","last_became_active":"2024-04-25T14:03:23.467848+0000","last_became_peered":"2024-04-25T14:03:23.467848+0000","last_unstale":"2024-04-25T14:03:24.472686+0000","last_undegraded":"2024-04-25T14:03:24.472686+0000","last_fullsized":"2024-04-25T14:03:24.472686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627126+0000","last_change":"2024-04-25T14:03:23.474747+0000","last_active":"2024-04-25T14:03:25.627126+0000","last_peered":"2024-04-25T14:03:25.627126+0000","last_clean":"2024-04-25T14:03:25.627126+0000","last_became_active":"2024-04-25T14:03:23.474447+0000","last_became_peered":"2024-04-25T14:03:23.474447+0000","last_unstale":"2024-04-25T14:03:25.627126+0000","last_undegraded":"2024-04-25T14:03:25.627126+0000","last_fullsized":"2024-04-25T14:03:25.627126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:38.809013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472104+0000","last_change":"2024-04-25T14:03:23.475499+0000","last_active":"2024-04-25T14:03:24.472104+0000","last_peered":"2024-04-25T14:03:24.472104+0000","last_clean":"2024-04-25T14:03:24.472104+0000","last_became_active":"2024-04-25T14:03:23.475156+0000","last_became_peered":"2024-04-25T14:03:23.475156+0000","last_unstale":"2024-04-25T14:03:24.472104+0000","last_undegraded":"2024-04-25T14:03:24.472104+0000","last_fullsized":"2024-04-25T14:03:24.472104+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:26.702517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627986+0000","last_change":"2024-04-25T14:03:22.458831+0000","last_active":"2024-04-25T14:03:25.627986+0000","last_peered":"2024-04-25T14:03:25.627986+0000","last_clean":"2024-04-25T14:03:25.627986+0000","last_became_active":"2024-04-25T14:03:22.458536+0000","last_became_peered":"2024-04-25T14:03:22.458536+0000","last_unstale":"2024-04-25T14:03:25.627986+0000","last_undegraded":"2024-04-25T14:03:25.627986+0000","last_fullsized":"2024-04-25T14:03:25.627986+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:40.514183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627737+0000","last_change":"2024-04-25T14:03:23.478517+0000","last_active":"2024-04-25T14:03:25.627737+0000","last_peered":"2024-04-25T14:03:25.627737+0000","last_clean":"2024-04-25T14:03:25.627737+0000","last_became_active":"2024-04-25T14:03:23.478062+0000","last_became_peered":"2024-04-25T14:03:23.478062+0000","last_unstale":"2024-04-25T14:03:25.627737+0000","last_undegraded":"2024-04-25T14:03:25.627737+0000","last_fullsized":"2024-04-25T14:03:25.627737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:59.589691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526572+0000","last_change":"2024-04-25T14:03:22.459437+0000","last_active":"2024-04-25T14:03:24.526572+0000","last_peered":"2024-04-25T14:03:24.526572+0000","last_clean":"2024-04-25T14:03:24.526572+0000","last_became_active":"2024-04-25T14:03:22.459099+0000","last_became_peered":"2024-04-25T14:03:22.459099+0000","last_unstale":"2024-04-25T14:03:24.526572+0000","last_undegraded":"2024-04-25T14:03:24.526572+0000","last_fullsized":"2024-04-25T14:03:24.526572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:34.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472787+0000","last_change":"2024-04-25T14:03:23.476910+0000","last_active":"2024-04-25T14:03:24.472787+0000","last_peered":"2024-04-25T14:03:24.472787+0000","last_clean":"2024-04-25T14:03:24.472787+0000","last_became_active":"2024-04-25T14:03:23.476338+0000","last_became_peered":"2024-04-25T14:03:23.476338+0000","last_unstale":"2024-04-25T14:03:24.472787+0000","last_undegraded":"2024-04-25T14:03:24.472787+0000","last_fullsized":"2024-04-25T14:03:24.472787+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:38.809013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627785+0000","last_change":"2024-04-25T14:03:22.456113+0000","last_active":"2024-04-25T14:03:25.627785+0000","last_peered":"2024-04-25T14:03:25.627785+0000","last_clean":"2024-04-25T14:03:25.627785+0000","last_became_active":"2024-04-25T14:03:22.455163+0000","last_became_peered":"2024-04-25T14:03:22.455163+0000","last_unstale":"2024-04-25T14:03:25.627785+0000","last_undegraded":"2024-04-25T14:03:25.627785+0000","last_fullsized":"2024-04-25T14:03:25.627785+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627453+0000","last_change":"2024-04-25T14:03:23.471660+0000","last_active":"2024-04-25T14:03:25.627453+0000","last_peered":"2024-04-25T14:03:25.627453+0000","last_clean":"2024-04-25T14:03:25.627453+0000","last_became_active":"2024-04-25T14:03:23.471344+0000","last_became_peered":"2024-04-25T14:03:23.471344+0000","last_unstale":"2024-04-25T14:03:25.627453+0000","last_undegraded":"2024-04-25T14:03:25.627453+0000","last_fullsized":"2024-04-25T14:03:25.627453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:58.416724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472078+0000","last_change":"2024-04-25T14:03:22.465365+0000","last_active":"2024-04-25T14:03:24.472078+0000","last_peered":"2024-04-25T14:03:24.472078+0000","last_clean":"2024-04-25T14:03:24.472078+0000","last_became_active":"2024-04-25T14:03:22.465127+0000","last_became_peered":"2024-04-25T14:03:22.465127+0000","last_unstale":"2024-04-25T14:03:24.472078+0000","last_undegraded":"2024-04-25T14:03:24.472078+0000","last_fullsized":"2024-04-25T14:03:24.472078+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:34.447246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526471+0000","last_change":"2024-04-25T14:03:23.468884+0000","last_active":"2024-04-25T14:03:24.526471+0000","last_peered":"2024-04-25T14:03:24.526471+0000","last_clean":"2024-04-25T14:03:24.526471+0000","last_became_active":"2024-04-25T14:03:23.468584+0000","last_became_peered":"2024-04-25T14:03:23.468584+0000","last_unstale":"2024-04-25T14:03:24.526471+0000","last_undegraded":"2024-04-25T14:03:24.526471+0000","last_fullsized":"2024-04-25T14:03:24.526471+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:59.589691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473356+0000","last_change":"2024-04-25T14:03:22.467405+0000","last_active":"2024-04-25T14:03:24.473356+0000","last_peered":"2024-04-25T14:03:24.473356+0000","last_clean":"2024-04-25T14:03:24.473356+0000","last_became_active":"2024-04-25T14:03:22.467057+0000","last_became_peered":"2024-04-25T14:03:22.467057+0000","last_unstale":"2024-04-25T14:03:24.473356+0000","last_undegraded":"2024-04-25T14:03:24.473356+0000","last_fullsized":"2024-04-25T14:03:24.473356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:03.552759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473530+0000","last_change":"2024-04-25T14:03:23.477620+0000","last_active":"2024-04-25T14:03:24.473530+0000","last_peered":"2024-04-25T14:03:24.473530+0000","last_clean":"2024-04-25T14:03:24.473530+0000","last_became_active":"2024-04-25T14:03:23.475898+0000","last_became_peered":"2024-04-25T14:03:23.475898+0000","last_unstale":"2024-04-25T14:03:24.473530+0000","last_undegraded":"2024-04-25T14:03:24.473530+0000","last_fullsized":"2024-04-25T14:03:24.473530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:17.109982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471267+0000","last_change":"2024-04-25T14:03:22.456091+0000","last_active":"2024-04-25T14:03:24.471267+0000","last_peered":"2024-04-25T14:03:24.471267+0000","last_clean":"2024-04-25T14:03:24.471267+0000","last_became_active":"2024-04-25T14:03:22.455732+0000","last_became_peered":"2024-04-25T14:03:22.455732+0000","last_unstale":"2024-04-25T14:03:24.471267+0000","last_undegraded":"2024-04-25T14:03:24.471267+0000","last_fullsized":"2024-04-25T14:03:24.471267+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:34.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472654+0000","last_change":"2024-04-25T14:03:23.476806+0000","last_active":"2024-04-25T14:03:24.472654+0000","last_peered":"2024-04-25T14:03:24.472654+0000","last_clean":"2024-04-25T14:03:24.472654+0000","last_became_active":"2024-04-25T14:03:23.476485+0000","last_became_peered":"2024-04-25T14:03:23.476485+0000","last_unstale":"2024-04-25T14:03:24.472654+0000","last_undegraded":"2024-04-25T14:03:24.472654+0000","last_fullsized":"2024-04-25T14:03:24.472654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627014+0000","last_change":"2024-04-25T14:03:22.464725+0000","last_active":"2024-04-25T14:03:25.627014+0000","last_peered":"2024-04-25T14:03:25.627014+0000","last_clean":"2024-04-25T14:03:25.627014+0000","last_became_active":"2024-04-25T14:03:22.464409+0000","last_became_peered":"2024-04-25T14:03:22.464409+0000","last_unstale":"2024-04-25T14:03:25.627014+0000","last_undegraded":"2024-04-25T14:03:25.627014+0000","last_fullsized":"2024-04-25T14:03:25.627014+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:40.729003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627908+0000","last_change":"2024-04-25T14:03:23.468975+0000","last_active":"2024-04-25T14:03:25.627908+0000","last_peered":"2024-04-25T14:03:25.627908+0000","last_clean":"2024-04-25T14:03:25.627908+0000","last_became_active":"2024-04-25T14:03:23.468661+0000","last_became_peered":"2024-04-25T14:03:23.468661+0000","last_unstale":"2024-04-25T14:03:25.627908+0000","last_undegraded":"2024-04-25T14:03:25.627908+0000","last_fullsized":"2024-04-25T14:03:25.627908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:18.678250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471332+0000","last_change":"2024-04-25T14:03:22.453661+0000","last_active":"2024-04-25T14:03:24.471332+0000","last_peered":"2024-04-25T14:03:24.471332+0000","last_clean":"2024-04-25T14:03:24.471332+0000","last_became_active":"2024-04-25T14:03:22.453283+0000","last_became_peered":"2024-04-25T14:03:22.453283+0000","last_unstale":"2024-04-25T14:03:24.471332+0000","last_undegraded":"2024-04-25T14:03:24.471332+0000","last_fullsized":"2024-04-25T14:03:24.471332+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:11.817679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473811+0000","last_change":"2024-04-25T14:03:23.481586+0000","last_active":"2024-04-25T14:03:24.473811+0000","last_peered":"2024-04-25T14:03:24.473811+0000","last_clean":"2024-04-25T14:03:24.473811+0000","last_became_active":"2024-04-25T14:03:23.481287+0000","last_became_peered":"2024-04-25T14:03:23.481287+0000","last_unstale":"2024-04-25T14:03:24.473811+0000","last_undegraded":"2024-04-25T14:03:24.473811+0000","last_fullsized":"2024-04-25T14:03:24.473811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:41.087596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627675+0000","last_change":"2024-04-25T14:03:22.463623+0000","last_active":"2024-04-25T14:03:25.627675+0000","last_peered":"2024-04-25T14:03:25.627675+0000","last_clean":"2024-04-25T14:03:25.627675+0000","last_became_active":"2024-04-25T14:03:22.463289+0000","last_became_peered":"2024-04-25T14:03:22.463289+0000","last_unstale":"2024-04-25T14:03:25.627675+0000","last_undegraded":"2024-04-25T14:03:25.627675+0000","last_fullsized":"2024-04-25T14:03:25.627675+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:37.802148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471665+0000","last_change":"2024-04-25T14:03:23.469389+0000","last_active":"2024-04-25T14:03:24.471665+0000","last_peered":"2024-04-25T14:03:24.471665+0000","last_clean":"2024-04-25T14:03:24.471665+0000","last_became_active":"2024-04-25T14:03:23.469087+0000","last_became_peered":"2024-04-25T14:03:23.469087+0000","last_unstale":"2024-04-25T14:03:24.471665+0000","last_undegraded":"2024-04-25T14:03:24.471665+0000","last_fullsized":"2024-04-25T14:03:24.471665+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:19.446529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471556+0000","last_change":"2024-04-25T14:03:22.468120+0000","last_active":"2024-04-25T14:03:24.471556+0000","last_peered":"2024-04-25T14:03:24.471556+0000","last_clean":"2024-04-25T14:03:24.471556+0000","last_became_active":"2024-04-25T14:03:22.467909+0000","last_became_peered":"2024-04-25T14:03:22.467909+0000","last_unstale":"2024-04-25T14:03:24.471556+0000","last_undegraded":"2024-04-25T14:03:24.471556+0000","last_fullsized":"2024-04-25T14:03:24.471556+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:02.306058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526771+0000","last_change":"2024-04-25T14:03:23.465564+0000","last_active":"2024-04-25T14:03:24.526771+0000","last_peered":"2024-04-25T14:03:24.526771+0000","last_clean":"2024-04-25T14:03:24.526771+0000","last_became_active":"2024-04-25T14:03:23.465258+0000","last_became_peered":"2024-04-25T14:03:23.465258+0000","last_unstale":"2024-04-25T14:03:24.526771+0000","last_undegraded":"2024-04-25T14:03:24.526771+0000","last_fullsized":"2024-04-25T14:03:24.526771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:58.295835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472585+0000","last_change":"2024-04-25T14:03:22.473042+0000","last_active":"2024-04-25T14:03:24.472585+0000","last_peered":"2024-04-25T14:03:24.472585+0000","last_clean":"2024-04-25T14:03:24.472585+0000","last_became_active":"2024-04-25T14:03:22.472772+0000","last_became_peered":"2024-04-25T14:03:22.472772+0000","last_unstale":"2024-04-25T14:03:24.472585+0000","last_undegraded":"2024-04-25T14:03:24.472585+0000","last_fullsized":"2024-04-25T14:03:24.472585+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:11.183503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473071+0000","last_change":"2024-04-25T14:03:23.478855+0000","last_active":"2024-04-25T14:03:24.473071+0000","last_peered":"2024-04-25T14:03:24.473071+0000","last_clean":"2024-04-25T14:03:24.473071+0000","last_became_active":"2024-04-25T14:03:23.478498+0000","last_became_peered":"2024-04-25T14:03:23.478498+0000","last_unstale":"2024-04-25T14:03:24.473071+0000","last_undegraded":"2024-04-25T14:03:24.473071+0000","last_fullsized":"2024-04-25T14:03:24.473071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:50.950138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627684+0000","last_change":"2024-04-25T14:03:22.465174+0000","last_active":"2024-04-25T14:03:25.627684+0000","last_peered":"2024-04-25T14:03:25.627684+0000","last_clean":"2024-04-25T14:03:25.627684+0000","last_became_active":"2024-04-25T14:03:22.464901+0000","last_became_peered":"2024-04-25T14:03:22.464901+0000","last_unstale":"2024-04-25T14:03:25.627684+0000","last_undegraded":"2024-04-25T14:03:25.627684+0000","last_fullsized":"2024-04-25T14:03:25.627684+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:11.183503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526491+0000","last_change":"2024-04-25T14:03:23.475200+0000","last_active":"2024-04-25T14:03:24.526491+0000","last_peered":"2024-04-25T14:03:24.526491+0000","last_clean":"2024-04-25T14:03:24.526491+0000","last_became_active":"2024-04-25T14:03:23.474922+0000","last_became_peered":"2024-04-25T14:03:23.474922+0000","last_unstale":"2024-04-25T14:03:24.526491+0000","last_undegraded":"2024-04-25T14:03:24.526491+0000","last_fullsized":"2024-04-25T14:03:24.526491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:03.870090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627877+0000","last_change":"2024-04-25T14:03:22.466901+0000","last_active":"2024-04-25T14:03:25.627877+0000","last_peered":"2024-04-25T14:03:25.627877+0000","last_clean":"2024-04-25T14:03:25.627877+0000","last_became_active":"2024-04-25T14:03:22.466600+0000","last_became_peered":"2024-04-25T14:03:22.466600+0000","last_unstale":"2024-04-25T14:03:25.627877+0000","last_undegraded":"2024-04-25T14:03:25.627877+0000","last_fullsized":"2024-04-25T14:03:25.627877+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:03.552759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627181+0000","last_change":"2024-04-25T14:03:23.473427+0000","last_active":"2024-04-25T14:03:25.627181+0000","last_peered":"2024-04-25T14:03:25.627181+0000","last_clean":"2024-04-25T14:03:25.627181+0000","last_became_active":"2024-04-25T14:03:23.473142+0000","last_became_peered":"2024-04-25T14:03:23.473142+0000","last_unstale":"2024-04-25T14:03:25.627181+0000","last_undegraded":"2024-04-25T14:03:25.627181+0000","last_fullsized":"2024-04-25T14:03:25.627181+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627294+0000","last_change":"2024-04-25T14:03:22.465444+0000","last_active":"2024-04-25T14:03:25.627294+0000","last_peered":"2024-04-25T14:03:25.627294+0000","last_clean":"2024-04-25T14:03:25.627294+0000","last_became_active":"2024-04-25T14:03:22.465099+0000","last_became_peered":"2024-04-25T14:03:22.465099+0000","last_unstale":"2024-04-25T14:03:25.627294+0000","last_undegraded":"2024-04-25T14:03:25.627294+0000","last_fullsized":"2024-04-25T14:03:25.627294+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:07.045482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471175+0000","last_change":"2024-04-25T14:03:23.468202+0000","last_active":"2024-04-25T14:03:24.471175+0000","last_peered":"2024-04-25T14:03:24.471175+0000","last_clean":"2024-04-25T14:03:24.471175+0000","last_became_active":"2024-04-25T14:03:23.467142+0000","last_became_peered":"2024-04-25T14:03:23.467142+0000","last_unstale":"2024-04-25T14:03:24.471175+0000","last_undegraded":"2024-04-25T14:03:24.471175+0000","last_fullsized":"2024-04-25T14:03:24.471175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:58.295835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627522+0000","last_change":"2024-04-25T14:03:22.458534+0000","last_active":"2024-04-25T14:03:25.627522+0000","last_peered":"2024-04-25T14:03:25.627522+0000","last_clean":"2024-04-25T14:03:25.627522+0000","last_became_active":"2024-04-25T14:03:22.458229+0000","last_became_peered":"2024-04-25T14:03:22.458229+0000","last_unstale":"2024-04-25T14:03:25.627522+0000","last_undegraded":"2024-04-25T14:03:25.627522+0000","last_fullsized":"2024-04-25T14:03:25.627522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:15.794799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627299+0000","last_change":"2024-04-25T14:03:23.468476+0000","last_active":"2024-04-25T14:03:25.627299+0000","last_peered":"2024-04-25T14:03:25.627299+0000","last_clean":"2024-04-25T14:03:25.627299+0000","last_became_active":"2024-04-25T14:03:23.467830+0000","last_became_peered":"2024-04-25T14:03:23.467830+0000","last_unstale":"2024-04-25T14:03:25.627299+0000","last_undegraded":"2024-04-25T14:03:25.627299+0000","last_fullsized":"2024-04-25T14:03:25.627299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471170+0000","last_change":"2024-04-25T14:03:22.457933+0000","last_active":"2024-04-25T14:03:24.471170+0000","last_peered":"2024-04-25T14:03:24.471170+0000","last_clean":"2024-04-25T14:03:24.471170+0000","last_became_active":"2024-04-25T14:03:22.457548+0000","last_became_peered":"2024-04-25T14:03:22.457548+0000","last_unstale":"2024-04-25T14:03:24.471170+0000","last_undegraded":"2024-04-25T14:03:24.471170+0000","last_fullsized":"2024-04-25T14:03:24.471170+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627641+0000","last_change":"2024-04-25T14:03:23.472401+0000","last_active":"2024-04-25T14:03:25.627641+0000","last_peered":"2024-04-25T14:03:25.627641+0000","last_clean":"2024-04-25T14:03:25.627641+0000","last_became_active":"2024-04-25T14:03:23.472116+0000","last_became_peered":"2024-04-25T14:03:23.472116+0000","last_unstale":"2024-04-25T14:03:25.627641+0000","last_undegraded":"2024-04-25T14:03:25.627641+0000","last_fullsized":"2024-04-25T14:03:25.627641+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:17.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473428+0000","last_change":"2024-04-25T14:03:22.470135+0000","last_active":"2024-04-25T14:03:24.473428+0000","last_peered":"2024-04-25T14:03:24.473428+0000","last_clean":"2024-04-25T14:03:24.473428+0000","last_became_active":"2024-04-25T14:03:22.469798+0000","last_became_peered":"2024-04-25T14:03:22.469798+0000","last_unstale":"2024-04-25T14:03:24.473428+0000","last_undegraded":"2024-04-25T14:03:24.473428+0000","last_fullsized":"2024-04-25T14:03:24.473428+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:43.711638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472452+0000","last_change":"2024-04-25T14:03:23.473341+0000","last_active":"2024-04-25T14:03:24.472452+0000","last_peered":"2024-04-25T14:03:24.472452+0000","last_clean":"2024-04-25T14:03:24.472452+0000","last_became_active":"2024-04-25T14:03:23.473018+0000","last_became_peered":"2024-04-25T14:03:23.473018+0000","last_unstale":"2024-04-25T14:03:24.472452+0000","last_undegraded":"2024-04-25T14:03:24.472452+0000","last_fullsized":"2024-04-25T14:03:24.472452+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472690+0000","last_change":"2024-04-25T14:03:22.462735+0000","last_active":"2024-04-25T14:03:24.472690+0000","last_peered":"2024-04-25T14:03:24.472690+0000","last_clean":"2024-04-25T14:03:24.472690+0000","last_became_active":"2024-04-25T14:03:22.462427+0000","last_became_peered":"2024-04-25T14:03:22.462427+0000","last_unstale":"2024-04-25T14:03:24.472690+0000","last_undegraded":"2024-04-25T14:03:24.472690+0000","last_fullsized":"2024-04-25T14:03:24.472690+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:40.729003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472525+0000","last_change":"2024-04-25T14:03:23.472230+0000","last_active":"2024-04-25T14:03:24.472525+0000","last_peered":"2024-04-25T14:03:24.472525+0000","last_clean":"2024-04-25T14:03:24.472525+0000","last_became_active":"2024-04-25T14:03:23.471916+0000","last_became_peered":"2024-04-25T14:03:23.471916+0000","last_unstale":"2024-04-25T14:03:24.472525+0000","last_undegraded":"2024-04-25T14:03:24.472525+0000","last_fullsized":"2024-04-25T14:03:24.472525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:44.718503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473747+0000","last_change":"2024-04-25T14:03:22.457434+0000","last_active":"2024-04-25T14:03:24.473747+0000","last_peered":"2024-04-25T14:03:24.473747+0000","last_clean":"2024-04-25T14:03:24.473747+0000","last_became_active":"2024-04-25T14:03:22.457122+0000","last_became_peered":"2024-04-25T14:03:22.457122+0000","last_unstale":"2024-04-25T14:03:24.473747+0000","last_undegraded":"2024-04-25T14:03:24.473747+0000","last_fullsized":"2024-04-25T14:03:24.473747+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:15.859362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473865+0000","last_change":"2024-04-25T14:03:23.470502+0000","last_active":"2024-04-25T14:03:24.473865+0000","last_peered":"2024-04-25T14:03:24.473865+0000","last_clean":"2024-04-25T14:03:24.473865+0000","last_became_active":"2024-04-25T14:03:23.470197+0000","last_became_peered":"2024-04-25T14:03:23.470197+0000","last_unstale":"2024-04-25T14:03:24.473865+0000","last_undegraded":"2024-04-25T14:03:24.473865+0000","last_fullsized":"2024-04-25T14:03:24.473865+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:35.454111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627397+0000","last_change":"2024-04-25T14:03:22.459335+0000","last_active":"2024-04-25T14:03:25.627397+0000","last_peered":"2024-04-25T14:03:25.627397+0000","last_clean":"2024-04-25T14:03:25.627397+0000","last_became_active":"2024-04-25T14:03:22.459008+0000","last_became_peered":"2024-04-25T14:03:22.459008+0000","last_unstale":"2024-04-25T14:03:25.627397+0000","last_undegraded":"2024-04-25T14:03:25.627397+0000","last_fullsized":"2024-04-25T14:03:25.627397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:40.496688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471399+0000","last_change":"2024-04-25T14:03:23.467432+0000","last_active":"2024-04-25T14:03:24.471399+0000","last_peered":"2024-04-25T14:03:24.471399+0000","last_clean":"2024-04-25T14:03:24.471399+0000","last_became_active":"2024-04-25T14:03:23.467100+0000","last_became_peered":"2024-04-25T14:03:23.467100+0000","last_unstale":"2024-04-25T14:03:24.471399+0000","last_undegraded":"2024-04-25T14:03:24.471399+0000","last_fullsized":"2024-04-25T14:03:24.471399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:03.945722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473464+0000","last_change":"2024-04-25T14:03:22.463630+0000","last_active":"2024-04-25T14:03:24.473464+0000","last_peered":"2024-04-25T14:03:24.473464+0000","last_clean":"2024-04-25T14:03:24.473464+0000","last_became_active":"2024-04-25T14:03:22.463311+0000","last_became_peered":"2024-04-25T14:03:22.463311+0000","last_unstale":"2024-04-25T14:03:24.473464+0000","last_undegraded":"2024-04-25T14:03:24.473464+0000","last_fullsized":"2024-04-25T14:03:24.473464+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:50.682001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526551+0000","last_change":"2024-04-25T14:03:23.470162+0000","last_active":"2024-04-25T14:03:24.526551+0000","last_peered":"2024-04-25T14:03:24.526551+0000","last_clean":"2024-04-25T14:03:24.526551+0000","last_became_active":"2024-04-25T14:03:23.469900+0000","last_became_peered":"2024-04-25T14:03:23.469900+0000","last_unstale":"2024-04-25T14:03:24.526551+0000","last_undegraded":"2024-04-25T14:03:24.526551+0000","last_fullsized":"2024-04-25T14:03:24.526551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:58.837156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473657+0000","last_change":"2024-04-25T14:03:22.455477+0000","last_active":"2024-04-25T14:03:24.473657+0000","last_peered":"2024-04-25T14:03:24.473657+0000","last_clean":"2024-04-25T14:03:24.473657+0000","last_became_active":"2024-04-25T14:03:22.455138+0000","last_became_peered":"2024-04-25T14:03:22.455138+0000","last_unstale":"2024-04-25T14:03:24.473657+0000","last_undegraded":"2024-04-25T14:03:24.473657+0000","last_fullsized":"2024-04-25T14:03:24.473657+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:34.353549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473226+0000","last_change":"2024-04-25T14:03:23.478492+0000","last_active":"2024-04-25T14:03:24.473226+0000","last_peered":"2024-04-25T14:03:24.473226+0000","last_clean":"2024-04-25T14:03:24.473226+0000","last_became_active":"2024-04-25T14:03:23.478120+0000","last_became_peered":"2024-04-25T14:03:23.478120+0000","last_unstale":"2024-04-25T14:03:24.473226+0000","last_undegraded":"2024-04-25T14:03:24.473226+0000","last_fullsized":"2024-04-25T14:03:24.473226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:58.416724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472752+0000","last_change":"2024-04-25T14:03:22.459674+0000","last_active":"2024-04-25T14:03:24.472752+0000","last_peered":"2024-04-25T14:03:24.472752+0000","last_clean":"2024-04-25T14:03:24.472752+0000","last_became_active":"2024-04-25T14:03:22.459373+0000","last_became_peered":"2024-04-25T14:03:22.459373+0000","last_unstale":"2024-04-25T14:03:24.472752+0000","last_undegraded":"2024-04-25T14:03:24.472752+0000","last_fullsized":"2024-04-25T14:03:24.472752+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:45:47.626684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627821+0000","last_change":"2024-04-25T14:03:23.468724+0000","last_active":"2024-04-25T14:03:25.627821+0000","last_peered":"2024-04-25T14:03:25.627821+0000","last_clean":"2024-04-25T14:03:25.627821+0000","last_became_active":"2024-04-25T14:03:23.468222+0000","last_became_peered":"2024-04-25T14:03:23.468222+0000","last_unstale":"2024-04-25T14:03:25.627821+0000","last_undegraded":"2024-04-25T14:03:25.627821+0000","last_fullsized":"2024-04-25T14:03:25.627821+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:46:25.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473802+0000","last_change":"2024-04-25T14:03:22.462280+0000","last_active":"2024-04-25T14:03:24.473802+0000","last_peered":"2024-04-25T14:03:24.473802+0000","last_clean":"2024-04-25T14:03:24.473802+0000","last_became_active":"2024-04-25T14:03:22.461941+0000","last_became_peered":"2024-04-25T14:03:22.461941+0000","last_unstale":"2024-04-25T14:03:24.473802+0000","last_undegraded":"2024-04-25T14:03:24.473802+0000","last_fullsized":"2024-04-25T14:03:24.473802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:08.832919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627436+0000","last_change":"2024-04-25T14:03:23.478576+0000","last_active":"2024-04-25T14:03:25.627436+0000","last_peered":"2024-04-25T14:03:25.627436+0000","last_clean":"2024-04-25T14:03:25.627436+0000","last_became_active":"2024-04-25T14:03:23.478257+0000","last_became_peered":"2024-04-25T14:03:23.478257+0000","last_unstale":"2024-04-25T14:03:25.627436+0000","last_undegraded":"2024-04-25T14:03:25.627436+0000","last_fullsized":"2024-04-25T14:03:25.627436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:30.179527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473547+0000","last_change":"2024-04-25T14:03:22.472916+0000","last_active":"2024-04-25T14:03:24.473547+0000","last_peered":"2024-04-25T14:03:24.473547+0000","last_clean":"2024-04-25T14:03:24.473547+0000","last_became_active":"2024-04-25T14:03:22.472691+0000","last_became_peered":"2024-04-25T14:03:22.472691+0000","last_unstale":"2024-04-25T14:03:24.473547+0000","last_undegraded":"2024-04-25T14:03:24.473547+0000","last_fullsized":"2024-04-25T14:03:24.473547+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:37.802148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471839+0000","last_change":"2024-04-25T14:03:23.467891+0000","last_active":"2024-04-25T14:03:24.471839+0000","last_peered":"2024-04-25T14:03:24.471839+0000","last_clean":"2024-04-25T14:03:24.471839+0000","last_became_active":"2024-04-25T14:03:23.467578+0000","last_became_peered":"2024-04-25T14:03:23.467578+0000","last_unstale":"2024-04-25T14:03:24.471839+0000","last_undegraded":"2024-04-25T14:03:24.471839+0000","last_fullsized":"2024-04-25T14:03:24.471839+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:28.888914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473870+0000","last_change":"2024-04-25T14:03:22.466332+0000","last_active":"2024-04-25T14:03:24.473870+0000","last_peered":"2024-04-25T14:03:24.473870+0000","last_clean":"2024-04-25T14:03:24.473870+0000","last_became_active":"2024-04-25T14:03:22.466032+0000","last_became_peered":"2024-04-25T14:03:22.466032+0000","last_unstale":"2024-04-25T14:03:24.473870+0000","last_undegraded":"2024-04-25T14:03:24.473870+0000","last_fullsized":"2024-04-25T14:03:24.473870+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:50.091226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627592+0000","last_change":"2024-04-25T14:03:23.475055+0000","last_active":"2024-04-25T14:03:25.627592+0000","last_peered":"2024-04-25T14:03:25.627592+0000","last_clean":"2024-04-25T14:03:25.627592+0000","last_became_active":"2024-04-25T14:03:23.474733+0000","last_became_peered":"2024-04-25T14:03:23.474733+0000","last_unstale":"2024-04-25T14:03:25.627592+0000","last_undegraded":"2024-04-25T14:03:25.627592+0000","last_fullsized":"2024-04-25T14:03:25.627592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:06:39.710834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526620+0000","last_change":"2024-04-25T14:03:22.467520+0000","last_active":"2024-04-25T14:03:24.526620+0000","last_peered":"2024-04-25T14:03:24.526620+0000","last_clean":"2024-04-25T14:03:24.526620+0000","last_became_active":"2024-04-25T14:03:22.467223+0000","last_became_peered":"2024-04-25T14:03:22.467223+0000","last_unstale":"2024-04-25T14:03:24.526620+0000","last_undegraded":"2024-04-25T14:03:24.526620+0000","last_fullsized":"2024-04-25T14:03:24.526620+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526775+0000","last_change":"2024-04-25T14:03:23.466245+0000","last_active":"2024-04-25T14:03:24.526775+0000","last_peered":"2024-04-25T14:03:24.526775+0000","last_clean":"2024-04-25T14:03:24.526775+0000","last_became_active":"2024-04-25T14:03:23.465975+0000","last_became_peered":"2024-04-25T14:03:23.465975+0000","last_unstale":"2024-04-25T14:03:24.526775+0000","last_undegraded":"2024-04-25T14:03:24.526775+0000","last_fullsized":"2024-04-25T14:03:24.526775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:03.945722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526745+0000","last_change":"2024-04-25T14:03:22.467232+0000","last_active":"2024-04-25T14:03:24.526745+0000","last_peered":"2024-04-25T14:03:24.526745+0000","last_clean":"2024-04-25T14:03:24.526745+0000","last_became_active":"2024-04-25T14:03:22.466908+0000","last_became_peered":"2024-04-25T14:03:22.466908+0000","last_unstale":"2024-04-25T14:03:24.526745+0000","last_undegraded":"2024-04-25T14:03:24.526745+0000","last_fullsized":"2024-04-25T14:03:24.526745+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.764344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627113+0000","last_change":"2024-04-25T14:03:23.468220+0000","last_active":"2024-04-25T14:03:25.627113+0000","last_peered":"2024-04-25T14:03:25.627113+0000","last_clean":"2024-04-25T14:03:25.627113+0000","last_became_active":"2024-04-25T14:03:23.467815+0000","last_became_peered":"2024-04-25T14:03:23.467815+0000","last_unstale":"2024-04-25T14:03:25.627113+0000","last_undegraded":"2024-04-25T14:03:25.627113+0000","last_fullsized":"2024-04-25T14:03:25.627113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627705+0000","last_change":"2024-04-25T14:03:22.460680+0000","last_active":"2024-04-25T14:03:25.627705+0000","last_peered":"2024-04-25T14:03:25.627705+0000","last_clean":"2024-04-25T14:03:25.627705+0000","last_became_active":"2024-04-25T14:03:22.460354+0000","last_became_peered":"2024-04-25T14:03:22.460354+0000","last_unstale":"2024-04-25T14:03:25.627705+0000","last_undegraded":"2024-04-25T14:03:25.627705+0000","last_fullsized":"2024-04-25T14:03:25.627705+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:11.817679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471885+0000","last_change":"2024-04-25T14:03:23.478976+0000","last_active":"2024-04-25T14:03:24.471885+0000","last_peered":"2024-04-25T14:03:24.471885+0000","last_clean":"2024-04-25T14:03:24.471885+0000","last_became_active":"2024-04-25T14:03:23.478509+0000","last_became_peered":"2024-04-25T14:03:23.478509+0000","last_unstale":"2024-04-25T14:03:24.471885+0000","last_undegraded":"2024-04-25T14:03:24.471885+0000","last_fullsized":"2024-04-25T14:03:24.471885+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:03.312923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.470832+0000","last_change":"2024-04-25T14:03:22.465180+0000","last_active":"2024-04-25T14:03:24.470832+0000","last_peered":"2024-04-25T14:03:24.470832+0000","last_clean":"2024-04-25T14:03:24.470832+0000","last_became_active":"2024-04-25T14:03:22.464890+0000","last_became_peered":"2024-04-25T14:03:22.464890+0000","last_unstale":"2024-04-25T14:03:24.470832+0000","last_undegraded":"2024-04-25T14:03:24.470832+0000","last_fullsized":"2024-04-25T14:03:24.470832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:40.514183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473200+0000","last_change":"2024-04-25T14:03:23.468642+0000","last_active":"2024-04-25T14:03:24.473200+0000","last_peered":"2024-04-25T14:03:24.473200+0000","last_clean":"2024-04-25T14:03:24.473200+0000","last_became_active":"2024-04-25T14:03:23.467883+0000","last_became_peered":"2024-04-25T14:03:23.467883+0000","last_unstale":"2024-04-25T14:03:24.473200+0000","last_undegraded":"2024-04-25T14:03:24.473200+0000","last_fullsized":"2024-04-25T14:03:24.473200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627187+0000","last_change":"2024-04-25T14:03:22.463281+0000","last_active":"2024-04-25T14:03:25.627187+0000","last_peered":"2024-04-25T14:03:25.627187+0000","last_clean":"2024-04-25T14:03:25.627187+0000","last_became_active":"2024-04-25T14:03:22.462977+0000","last_became_peered":"2024-04-25T14:03:22.462977+0000","last_unstale":"2024-04-25T14:03:25.627187+0000","last_undegraded":"2024-04-25T14:03:25.627187+0000","last_fullsized":"2024-04-25T14:03:25.627187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:43.711638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473469+0000","last_change":"2024-04-25T14:03:23.476670+0000","last_active":"2024-04-25T14:03:24.473469+0000","last_peered":"2024-04-25T14:03:24.473469+0000","last_clean":"2024-04-25T14:03:24.473469+0000","last_became_active":"2024-04-25T14:03:23.476307+0000","last_became_peered":"2024-04-25T14:03:23.476307+0000","last_unstale":"2024-04-25T14:03:24.473469+0000","last_undegraded":"2024-04-25T14:03:24.473469+0000","last_fullsized":"2024-04-25T14:03:24.473469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:18.678250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472168+0000","last_change":"2024-04-25T14:03:22.463970+0000","last_active":"2024-04-25T14:03:24.472168+0000","last_peered":"2024-04-25T14:03:24.472168+0000","last_clean":"2024-04-25T14:03:24.472168+0000","last_became_active":"2024-04-25T14:03:22.463642+0000","last_became_peered":"2024-04-25T14:03:22.463642+0000","last_unstale":"2024-04-25T14:03:24.472168+0000","last_undegraded":"2024-04-25T14:03:24.472168+0000","last_fullsized":"2024-04-25T14:03:24.472168+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.764344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":146,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471460+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:03:24.471460+0000","last_peered":"2024-04-25T14:03:24.471460+0000","last_clean":"2024-04-25T14:03:24.471460+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:03:24.471460+0000","last_undegraded":"2024-04-25T14:03:24.471460+0000","last_fullsized":"2024-04-25T14:03:24.471460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":782086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":190148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":190640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":191682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1036288,"data_stored":780438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:03:31.927 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:03:32.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.048+0000 7f4c007d7640 1 Processor -- start 2024-04-25T14:03:32.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.048+0000 7f4c007d7640 1 -- start start 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4c007d7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4bf80b3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4c007d7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4bf806cbf0 0x7f4bf806d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4c007d7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bf806d540 con 0x7f4bf80b2650 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4c007d7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4bf80b39b0 con 0x7f4bf806cbf0 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4c007d7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf8160570 con 0x7f4bf8160160 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfe573640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4bf80b2650 0x7f4bf80b74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34500/0 (socket says 172.21.15.150:34500) 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfe573640 1 -- 172.21.15.150:0/3672627089 learned_addr learned my addr 172.21.15.150:0/3672627089 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfed74640 1 --2- 172.21.15.150:0/3672627089 >> v2:172.21.15.187:3300/0 conn(0x7f4bf806cbf0 0x7f4bf806d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfdd72640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4bf80b3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4003367694 0 0) 0x7f4bf4001490 con 0x7f4bf80b2650 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4be4003520 con 0x7f4bf80b2650 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2830555464 0 0) 0x7f4be4003520 con 0x7f4bf80b2650 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 msgr2=0x7f4bf80b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4bf80b3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 >> v2:172.21.15.187:3300/0 conn(0x7f4bf806cbf0 msgr2=0x7f4bf806d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 --2- 172.21.15.150:0/3672627089 >> v2:172.21.15.187:3300/0 conn(0x7f4bf806cbf0 0x7f4bf806d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf80b2de0 con 0x7f4bf80b2650 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f4bf4001bc0 con 0x7f4bf80b2650 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.050+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 >> v1:172.21.15.150:6789/0 conn(0x7f4bf80b2650 legacy=0x7f4bf80b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4be4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be4004800 0x7f4be4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4be4005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf80b39b0 con 0x7f4be40039b0 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf8160570 con 0x7f4be4004800 2024-04-25T14:03:32.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4be400ba70 con 0x7f4bf8160160 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4be4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfdd72640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be4004800 0x7f4be4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfed74640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4be4005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 -- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 msgr2=0x7f4be4005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf8160160 0x7f4be4005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 -- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be4004800 msgr2=0x7f4be4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4be4004800 0x7f4be4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.051+0000 7f4bfe573640 1 -- 172.21.15.150:0/3672627089 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be4003520 con 0x7f4be40039b0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.052+0000 7f4bfe573640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4be4003e20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400aa50 tx=0x7f4be4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e88b7f809aa1ac7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.052+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400a670 con 0x7f4be40039b0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.052+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf402f070 con 0x7f4be40039b0 2024-04-25T14:03:32.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.052+0000 7f4bfd571640 1 -- 172.21.15.150:0/3672627089 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4034470 con 0x7f4be40039b0 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 -- 172.21.15.150:0/3672627089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 msgr2=0x7f4be4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 --2- 172.21.15.150:0/3672627089 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4be4003e20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4bf400aa50 tx=0x7f4be4005830 comp rx=0 tx=0).stop 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4bfed74640 1 -- 172.21.15.150:0/3672627089 reap_dead start 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 -- 172.21.15.150:0/3672627089 shutdown_connections 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 -- 172.21.15.150:0/3672627089 >> 172.21.15.150:0/3672627089 conn(0x7f4bf8073ce0 msgr2=0x7f4bf809b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:32.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 -- 172.21.15.150:0/3672627089 shutdown_connections 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.053+0000 7f4c007d7640 1 -- 172.21.15.150:0/3672627089 wait complete. 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.054+0000 7f4c007d7640 1 Processor -- start 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.055+0000 7f4c007d7640 1 -- start start 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bf8158370 0x7f4bf81557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf81545f0 0x7f4bf815cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf80b3640 con 0x7f4be40039b0 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf80b3360 con 0x7f4bf8158370 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4c007d7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4bf80b3e40 con 0x7f4bf81545f0 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45182/0 (socket says 172.21.15.150:45182) 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 -- 172.21.15.150:0/2808001221 learned_addr learned my addr 172.21.15.150:0/2808001221 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfdd72640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bf8158370 0x7f4bf81557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfed74640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf81545f0 0x7f4bf815cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 -- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf81545f0 msgr2=0x7f4bf815cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf81545f0 0x7f4bf815cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 -- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bf8158370 msgr2=0x7f4bf81557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bf8158370 0x7f4bf81557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 -- 172.21.15.150:0/2808001221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf80cd910 con 0x7f4be40039b0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4bfe573640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4bf402b720 tx=0x7f4bf81606f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.056+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf4044070 con 0x7f4be40039b0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.057+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf402ba70 con 0x7f4be40039b0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.057+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf80cdbf0 con 0x7f4be40039b0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.057+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf400a670 con 0x7f4be40039b0 2024-04-25T14:03:32.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.057+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf815c360 con 0x7f4be40039b0 2024-04-25T14:03:32.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.058+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf8076a40 con 0x7f4be40039b0 2024-04-25T14:03:32.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.058+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4bf4047070 con 0x7f4be40039b0 2024-04-25T14:03:32.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.059+0000 7f4beeffd640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 0x7f4bdc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.059+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f4bf40b5630 con 0x7f4be40039b0 2024-04-25T14:03:32.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.059+0000 7f4bfdd72640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 0x7f4bdc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.059+0000 7f4bfdd72640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 0x7f4bdc0769c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4bf815f660 tx=0x7f4be8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.064+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bf806d540 con 0x7f4be40039b0 2024-04-25T14:03:32.234 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.235+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4bf8155060 con 0x7f4bdc0744b0 2024-04-25T14:03:32.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.241+0000 7f4beeffd640 1 -- 172.21.15.150:0/2808001221 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251948 (secure 0 0 0) 0x7f4bf8155060 con 0x7f4bdc0744b0 2024-04-25T14:03:32.243 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:32.244 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 msgr2=0x7f4bdc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 0x7f4bdc0769c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f4bf815f660 tx=0x7f4be8008040 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 msgr2=0x7f4bf80cd0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4bf402b720 tx=0x7f4bf81606f0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 shutdown_connections 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4bdc0744b0 0x7f4bdc0769c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4bf81545f0 0x7f4bf815cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4bf8158370 0x7f4bf81557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 --2- 172.21.15.150:0/2808001221 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4be40039b0 0x7f4bf80cd0e0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 >> 172.21.15.150:0/2808001221 conn(0x7f4bf8073ce0 msgr2=0x7f4bf80b7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:32.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 shutdown_connections 2024-04-25T14:03:32.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.244+0000 7f4c007d7640 1 -- 172.21.15.150:0/2808001221 wait complete. 2024-04-25T14:03:32.255 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-25T14:03:30.604453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":146,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":111,"ondisk_log_size":111,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":187,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220816,"kb_used_data":4648,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753904,"statfs":{"total":773094113280,"available":772867997696,"internally_reserved":0,"allocated":4759552,"data_stored":2705844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001615"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627620+0000","last_change":"2024-04-25T14:03:23.474092+0000","last_active":"2024-04-25T14:03:25.627620+0000","last_peered":"2024-04-25T14:03:25.627620+0000","last_clean":"2024-04-25T14:03:25.627620+0000","last_became_active":"2024-04-25T14:03:23.473817+0000","last_became_peered":"2024-04-25T14:03:23.473817+0000","last_unstale":"2024-04-25T14:03:25.627620+0000","last_undegraded":"2024-04-25T14:03:25.627620+0000","last_fullsized":"2024-04-25T14:03:25.627620+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:59.615179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627424+0000","last_change":"2024-04-25T14:03:23.471063+0000","last_active":"2024-04-25T14:03:25.627424+0000","last_peered":"2024-04-25T14:03:25.627424+0000","last_clean":"2024-04-25T14:03:25.627424+0000","last_became_active":"2024-04-25T14:03:23.470801+0000","last_became_peered":"2024-04-25T14:03:23.470801+0000","last_unstale":"2024-04-25T14:03:25.627424+0000","last_undegraded":"2024-04-25T14:03:25.627424+0000","last_fullsized":"2024-04-25T14:03:25.627424+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526798+0000","last_change":"2024-04-25T14:03:23.475260+0000","last_active":"2024-04-25T14:03:24.526798+0000","last_peered":"2024-04-25T14:03:24.526798+0000","last_clean":"2024-04-25T14:03:24.526798+0000","last_became_active":"2024-04-25T14:03:23.474934+0000","last_became_peered":"2024-04-25T14:03:23.474934+0000","last_unstale":"2024-04-25T14:03:24.526798+0000","last_undegraded":"2024-04-25T14:03:24.526798+0000","last_fullsized":"2024-04-25T14:03:24.526798+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:19.446529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473756+0000","last_change":"2024-04-25T14:03:23.481483+0000","last_active":"2024-04-25T14:03:24.473756+0000","last_peered":"2024-04-25T14:03:24.473756+0000","last_clean":"2024-04-25T14:03:24.473756+0000","last_became_active":"2024-04-25T14:03:23.481184+0000","last_became_peered":"2024-04-25T14:03:23.481184+0000","last_unstale":"2024-04-25T14:03:24.473756+0000","last_undegraded":"2024-04-25T14:03:24.473756+0000","last_fullsized":"2024-04-25T14:03:24.473756+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:41.521048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473581+0000","last_change":"2024-04-25T14:03:23.480208+0000","last_active":"2024-04-25T14:03:24.473581+0000","last_peered":"2024-04-25T14:03:24.473581+0000","last_clean":"2024-04-25T14:03:24.473581+0000","last_became_active":"2024-04-25T14:03:23.479895+0000","last_became_peered":"2024-04-25T14:03:23.479895+0000","last_unstale":"2024-04-25T14:03:24.473581+0000","last_undegraded":"2024-04-25T14:03:24.473581+0000","last_fullsized":"2024-04-25T14:03:24.473581+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:12.824544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627237+0000","last_change":"2024-04-25T14:03:23.467851+0000","last_active":"2024-04-25T14:03:25.627237+0000","last_peered":"2024-04-25T14:03:25.627237+0000","last_clean":"2024-04-25T14:03:25.627237+0000","last_became_active":"2024-04-25T14:03:23.467550+0000","last_became_peered":"2024-04-25T14:03:23.467550+0000","last_unstale":"2024-04-25T14:03:25.627237+0000","last_undegraded":"2024-04-25T14:03:25.627237+0000","last_fullsized":"2024-04-25T14:03:25.627237+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:12.190368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473427+0000","last_change":"2024-04-25T14:03:23.467568+0000","last_active":"2024-04-25T14:03:24.473427+0000","last_peered":"2024-04-25T14:03:24.473427+0000","last_clean":"2024-04-25T14:03:24.473427+0000","last_became_active":"2024-04-25T14:03:23.467221+0000","last_became_peered":"2024-04-25T14:03:23.467221+0000","last_unstale":"2024-04-25T14:03:24.473427+0000","last_undegraded":"2024-04-25T14:03:24.473427+0000","last_fullsized":"2024-04-25T14:03:24.473427+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:17.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627126+0000","last_change":"2024-04-25T14:03:23.481082+0000","last_active":"2024-04-25T14:03:25.627126+0000","last_peered":"2024-04-25T14:03:25.627126+0000","last_clean":"2024-04-25T14:03:25.627126+0000","last_became_active":"2024-04-25T14:03:23.480782+0000","last_became_peered":"2024-04-25T14:03:23.480782+0000","last_unstale":"2024-04-25T14:03:25.627126+0000","last_undegraded":"2024-04-25T14:03:25.627126+0000","last_fullsized":"2024-04-25T14:03:25.627126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526852+0000","last_change":"2024-04-25T14:03:23.479982+0000","last_active":"2024-04-25T14:03:24.526852+0000","last_peered":"2024-04-25T14:03:24.526852+0000","last_clean":"2024-04-25T14:03:24.526852+0000","last_became_active":"2024-04-25T14:03:23.479685+0000","last_became_peered":"2024-04-25T14:03:23.479685+0000","last_unstale":"2024-04-25T14:03:24.526852+0000","last_undegraded":"2024-04-25T14:03:24.526852+0000","last_fullsized":"2024-04-25T14:03:24.526852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:28.888914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471339+0000","last_change":"2024-04-25T14:03:23.476848+0000","last_active":"2024-04-25T14:03:24.471339+0000","last_peered":"2024-04-25T14:03:24.471339+0000","last_clean":"2024-04-25T14:03:24.471339+0000","last_became_active":"2024-04-25T14:03:23.476533+0000","last_became_peered":"2024-04-25T14:03:23.476533+0000","last_unstale":"2024-04-25T14:03:24.471339+0000","last_undegraded":"2024-04-25T14:03:24.471339+0000","last_fullsized":"2024-04-25T14:03:24.471339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472057+0000","last_change":"2024-04-25T14:03:23.481025+0000","last_active":"2024-04-25T14:03:24.472057+0000","last_peered":"2024-04-25T14:03:24.472057+0000","last_clean":"2024-04-25T14:03:24.472057+0000","last_became_active":"2024-04-25T14:03:23.480714+0000","last_became_peered":"2024-04-25T14:03:23.480714+0000","last_unstale":"2024-04-25T14:03:24.472057+0000","last_undegraded":"2024-04-25T14:03:24.472057+0000","last_fullsized":"2024-04-25T14:03:24.472057+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526654+0000","last_change":"2024-04-25T14:03:23.473566+0000","last_active":"2024-04-25T14:03:24.526654+0000","last_peered":"2024-04-25T14:03:24.526654+0000","last_clean":"2024-04-25T14:03:24.526654+0000","last_became_active":"2024-04-25T14:03:23.473248+0000","last_became_peered":"2024-04-25T14:03:23.473248+0000","last_unstale":"2024-04-25T14:03:24.526654+0000","last_undegraded":"2024-04-25T14:03:24.526654+0000","last_fullsized":"2024-04-25T14:03:24.526654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:28.838095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473293+0000","last_change":"2024-04-25T14:03:23.469460+0000","last_active":"2024-04-25T14:03:24.473293+0000","last_peered":"2024-04-25T14:03:24.473293+0000","last_clean":"2024-04-25T14:03:24.473293+0000","last_became_active":"2024-04-25T14:03:23.469090+0000","last_became_peered":"2024-04-25T14:03:23.469090+0000","last_unstale":"2024-04-25T14:03:24.473293+0000","last_undegraded":"2024-04-25T14:03:24.473293+0000","last_fullsized":"2024-04-25T14:03:24.473293+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:06:39.710834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471662+0000","last_change":"2024-04-25T14:03:23.478945+0000","last_active":"2024-04-25T14:03:24.471662+0000","last_peered":"2024-04-25T14:03:24.471662+0000","last_clean":"2024-04-25T14:03:24.471662+0000","last_became_active":"2024-04-25T14:03:23.478600+0000","last_became_peered":"2024-04-25T14:03:23.478600+0000","last_unstale":"2024-04-25T14:03:24.471662+0000","last_undegraded":"2024-04-25T14:03:24.471662+0000","last_fullsized":"2024-04-25T14:03:24.471662+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:48.357470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473313+0000","last_change":"2024-04-25T14:03:23.476367+0000","last_active":"2024-04-25T14:03:24.473313+0000","last_peered":"2024-04-25T14:03:24.473313+0000","last_clean":"2024-04-25T14:03:24.473313+0000","last_became_active":"2024-04-25T14:03:23.476047+0000","last_became_peered":"2024-04-25T14:03:23.476047+0000","last_unstale":"2024-04-25T14:03:24.473313+0000","last_undegraded":"2024-04-25T14:03:24.473313+0000","last_fullsized":"2024-04-25T14:03:24.473313+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:06:59.615179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471103+0000","last_change":"2024-04-25T14:03:23.473411+0000","last_active":"2024-04-25T14:03:24.471103+0000","last_peered":"2024-04-25T14:03:24.471103+0000","last_clean":"2024-04-25T14:03:24.471103+0000","last_became_active":"2024-04-25T14:03:23.473087+0000","last_became_peered":"2024-04-25T14:03:23.473087+0000","last_unstale":"2024-04-25T14:03:24.471103+0000","last_undegraded":"2024-04-25T14:03:24.471103+0000","last_fullsized":"2024-04-25T14:03:24.471103+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:28.838095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473004+0000","last_change":"2024-04-25T14:03:23.479719+0000","last_active":"2024-04-25T14:03:24.473004+0000","last_peered":"2024-04-25T14:03:24.473004+0000","last_clean":"2024-04-25T14:03:24.473004+0000","last_became_active":"2024-04-25T14:03:23.479367+0000","last_became_peered":"2024-04-25T14:03:23.479367+0000","last_unstale":"2024-04-25T14:03:24.473004+0000","last_undegraded":"2024-04-25T14:03:24.473004+0000","last_fullsized":"2024-04-25T14:03:24.473004+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628123+0000","last_change":"2024-04-25T14:03:23.476703+0000","last_active":"2024-04-25T14:03:25.628123+0000","last_peered":"2024-04-25T14:03:25.628123+0000","last_clean":"2024-04-25T14:03:25.628123+0000","last_became_active":"2024-04-25T14:03:23.476391+0000","last_became_peered":"2024-04-25T14:03:23.476391+0000","last_unstale":"2024-04-25T14:03:25.628123+0000","last_undegraded":"2024-04-25T14:03:25.628123+0000","last_fullsized":"2024-04-25T14:03:25.628123+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473932+0000","last_change":"2024-04-25T14:03:23.476020+0000","last_active":"2024-04-25T14:03:24.473932+0000","last_peered":"2024-04-25T14:03:24.473932+0000","last_clean":"2024-04-25T14:03:24.473932+0000","last_became_active":"2024-04-25T14:03:23.475456+0000","last_became_peered":"2024-04-25T14:03:23.475456+0000","last_unstale":"2024-04-25T14:03:24.473932+0000","last_undegraded":"2024-04-25T14:03:24.473932+0000","last_fullsized":"2024-04-25T14:03:24.473932+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:41.735868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627968+0000","last_change":"2024-04-25T14:03:23.480347+0000","last_active":"2024-04-25T14:03:25.627968+0000","last_peered":"2024-04-25T14:03:25.627968+0000","last_clean":"2024-04-25T14:03:25.627968+0000","last_became_active":"2024-04-25T14:03:23.480007+0000","last_became_peered":"2024-04-25T14:03:23.480007+0000","last_unstale":"2024-04-25T14:03:25.627968+0000","last_undegraded":"2024-04-25T14:03:25.627968+0000","last_fullsized":"2024-04-25T14:03:25.627968+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:53:48.357470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473453+0000","last_change":"2024-04-25T14:03:23.477606+0000","last_active":"2024-04-25T14:03:24.473453+0000","last_peered":"2024-04-25T14:03:24.473453+0000","last_clean":"2024-04-25T14:03:24.473453+0000","last_became_active":"2024-04-25T14:03:23.475489+0000","last_became_peered":"2024-04-25T14:03:23.475489+0000","last_unstale":"2024-04-25T14:03:24.473453+0000","last_undegraded":"2024-04-25T14:03:24.473453+0000","last_fullsized":"2024-04-25T14:03:24.473453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:46:25.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472926+0000","last_change":"2024-04-25T14:03:23.477678+0000","last_active":"2024-04-25T14:03:24.472926+0000","last_peered":"2024-04-25T14:03:24.472926+0000","last_clean":"2024-04-25T14:03:24.472926+0000","last_became_active":"2024-04-25T14:03:23.477360+0000","last_became_peered":"2024-04-25T14:03:23.477360+0000","last_unstale":"2024-04-25T14:03:24.472926+0000","last_undegraded":"2024-04-25T14:03:24.472926+0000","last_fullsized":"2024-04-25T14:03:24.472926+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:12.190368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472493+0000","last_change":"2024-04-25T14:03:23.475106+0000","last_active":"2024-04-25T14:03:24.472493+0000","last_peered":"2024-04-25T14:03:24.472493+0000","last_clean":"2024-04-25T14:03:24.472493+0000","last_became_active":"2024-04-25T14:03:23.474775+0000","last_became_peered":"2024-04-25T14:03:23.474775+0000","last_unstale":"2024-04-25T14:03:24.472493+0000","last_undegraded":"2024-04-25T14:03:24.472493+0000","last_fullsized":"2024-04-25T14:03:24.472493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:42:56.217467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473560+0000","last_change":"2024-04-25T14:03:23.472420+0000","last_active":"2024-04-25T14:03:24.473560+0000","last_peered":"2024-04-25T14:03:24.473560+0000","last_clean":"2024-04-25T14:03:24.473560+0000","last_became_active":"2024-04-25T14:03:23.472051+0000","last_became_peered":"2024-04-25T14:03:23.472051+0000","last_unstale":"2024-04-25T14:03:24.473560+0000","last_undegraded":"2024-04-25T14:03:24.473560+0000","last_fullsized":"2024-04-25T14:03:24.473560+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:16.801664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628042+0000","last_change":"2024-04-25T14:03:23.477127+0000","last_active":"2024-04-25T14:03:25.628042+0000","last_peered":"2024-04-25T14:03:25.628042+0000","last_clean":"2024-04-25T14:03:25.628042+0000","last_became_active":"2024-04-25T14:03:23.473390+0000","last_became_peered":"2024-04-25T14:03:23.473390+0000","last_unstale":"2024-04-25T14:03:25.628042+0000","last_undegraded":"2024-04-25T14:03:25.628042+0000","last_fullsized":"2024-04-25T14:03:25.628042+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:44.718503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627397+0000","last_change":"2024-04-25T14:03:23.478890+0000","last_active":"2024-04-25T14:03:25.627397+0000","last_peered":"2024-04-25T14:03:25.627397+0000","last_clean":"2024-04-25T14:03:25.627397+0000","last_became_active":"2024-04-25T14:03:23.478468+0000","last_became_peered":"2024-04-25T14:03:23.478468+0000","last_unstale":"2024-04-25T14:03:25.627397+0000","last_undegraded":"2024-04-25T14:03:25.627397+0000","last_fullsized":"2024-04-25T14:03:25.627397+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:50.950138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.628023+0000","last_change":"2024-04-25T14:03:23.472415+0000","last_active":"2024-04-25T14:03:25.628023+0000","last_peered":"2024-04-25T14:03:25.628023+0000","last_clean":"2024-04-25T14:03:25.628023+0000","last_became_active":"2024-04-25T14:03:23.472112+0000","last_became_peered":"2024-04-25T14:03:23.472112+0000","last_unstale":"2024-04-25T14:03:25.628023+0000","last_undegraded":"2024-04-25T14:03:25.628023+0000","last_fullsized":"2024-04-25T14:03:25.628023+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:03.312923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471426+0000","last_change":"2024-04-25T14:03:23.473541+0000","last_active":"2024-04-25T14:03:24.471426+0000","last_peered":"2024-04-25T14:03:24.471426+0000","last_clean":"2024-04-25T14:03:24.471426+0000","last_became_active":"2024-04-25T14:03:23.473227+0000","last_became_peered":"2024-04-25T14:03:23.473227+0000","last_unstale":"2024-04-25T14:03:24.471426+0000","last_undegraded":"2024-04-25T14:03:24.471426+0000","last_fullsized":"2024-04-25T14:03:24.471426+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627818+0000","last_change":"2024-04-25T14:03:23.468805+0000","last_active":"2024-04-25T14:03:25.627818+0000","last_peered":"2024-04-25T14:03:25.627818+0000","last_clean":"2024-04-25T14:03:25.627818+0000","last_became_active":"2024-04-25T14:03:23.468502+0000","last_became_peered":"2024-04-25T14:03:23.468502+0000","last_unstale":"2024-04-25T14:03:25.627818+0000","last_undegraded":"2024-04-25T14:03:25.627818+0000","last_fullsized":"2024-04-25T14:03:25.627818+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:45:48.633548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627826+0000","last_change":"2024-04-25T14:03:23.469483+0000","last_active":"2024-04-25T14:03:25.627826+0000","last_peered":"2024-04-25T14:03:25.627826+0000","last_clean":"2024-04-25T14:03:25.627826+0000","last_became_active":"2024-04-25T14:03:23.469191+0000","last_became_peered":"2024-04-25T14:03:23.469191+0000","last_unstale":"2024-04-25T14:03:25.627826+0000","last_undegraded":"2024-04-25T14:03:25.627826+0000","last_fullsized":"2024-04-25T14:03:25.627826+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:40.771209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472686+0000","last_change":"2024-04-25T14:03:23.468203+0000","last_active":"2024-04-25T14:03:24.472686+0000","last_peered":"2024-04-25T14:03:24.472686+0000","last_clean":"2024-04-25T14:03:24.472686+0000","last_became_active":"2024-04-25T14:03:23.467848+0000","last_became_peered":"2024-04-25T14:03:23.467848+0000","last_unstale":"2024-04-25T14:03:24.472686+0000","last_undegraded":"2024-04-25T14:03:24.472686+0000","last_fullsized":"2024-04-25T14:03:24.472686+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627126+0000","last_change":"2024-04-25T14:03:23.474747+0000","last_active":"2024-04-25T14:03:25.627126+0000","last_peered":"2024-04-25T14:03:25.627126+0000","last_clean":"2024-04-25T14:03:25.627126+0000","last_became_active":"2024-04-25T14:03:23.474447+0000","last_became_peered":"2024-04-25T14:03:23.474447+0000","last_unstale":"2024-04-25T14:03:25.627126+0000","last_undegraded":"2024-04-25T14:03:25.627126+0000","last_fullsized":"2024-04-25T14:03:25.627126+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:38.809013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472104+0000","last_change":"2024-04-25T14:03:23.475499+0000","last_active":"2024-04-25T14:03:24.472104+0000","last_peered":"2024-04-25T14:03:24.472104+0000","last_clean":"2024-04-25T14:03:24.472104+0000","last_became_active":"2024-04-25T14:03:23.475156+0000","last_became_peered":"2024-04-25T14:03:23.475156+0000","last_unstale":"2024-04-25T14:03:24.472104+0000","last_undegraded":"2024-04-25T14:03:24.472104+0000","last_fullsized":"2024-04-25T14:03:24.472104+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:26.702517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"37'4","reported_seq":18,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627986+0000","last_change":"2024-04-25T14:03:22.458831+0000","last_active":"2024-04-25T14:03:25.627986+0000","last_peered":"2024-04-25T14:03:25.627986+0000","last_clean":"2024-04-25T14:03:25.627986+0000","last_became_active":"2024-04-25T14:03:22.458536+0000","last_became_peered":"2024-04-25T14:03:22.458536+0000","last_unstale":"2024-04-25T14:03:25.627986+0000","last_undegraded":"2024-04-25T14:03:25.627986+0000","last_fullsized":"2024-04-25T14:03:25.627986+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:40.514183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627737+0000","last_change":"2024-04-25T14:03:23.478517+0000","last_active":"2024-04-25T14:03:25.627737+0000","last_peered":"2024-04-25T14:03:25.627737+0000","last_clean":"2024-04-25T14:03:25.627737+0000","last_became_active":"2024-04-25T14:03:23.478062+0000","last_became_peered":"2024-04-25T14:03:23.478062+0000","last_unstale":"2024-04-25T14:03:25.627737+0000","last_undegraded":"2024-04-25T14:03:25.627737+0000","last_fullsized":"2024-04-25T14:03:25.627737+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:59.589691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526572+0000","last_change":"2024-04-25T14:03:22.459437+0000","last_active":"2024-04-25T14:03:24.526572+0000","last_peered":"2024-04-25T14:03:24.526572+0000","last_clean":"2024-04-25T14:03:24.526572+0000","last_became_active":"2024-04-25T14:03:22.459099+0000","last_became_peered":"2024-04-25T14:03:22.459099+0000","last_unstale":"2024-04-25T14:03:24.526572+0000","last_undegraded":"2024-04-25T14:03:24.526572+0000","last_fullsized":"2024-04-25T14:03:24.526572+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:34.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472787+0000","last_change":"2024-04-25T14:03:23.476910+0000","last_active":"2024-04-25T14:03:24.472787+0000","last_peered":"2024-04-25T14:03:24.472787+0000","last_clean":"2024-04-25T14:03:24.472787+0000","last_became_active":"2024-04-25T14:03:23.476338+0000","last_became_peered":"2024-04-25T14:03:23.476338+0000","last_unstale":"2024-04-25T14:03:24.472787+0000","last_undegraded":"2024-04-25T14:03:24.472787+0000","last_fullsized":"2024-04-25T14:03:24.472787+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:38.809013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627785+0000","last_change":"2024-04-25T14:03:22.456113+0000","last_active":"2024-04-25T14:03:25.627785+0000","last_peered":"2024-04-25T14:03:25.627785+0000","last_clean":"2024-04-25T14:03:25.627785+0000","last_became_active":"2024-04-25T14:03:22.455163+0000","last_became_peered":"2024-04-25T14:03:22.455163+0000","last_unstale":"2024-04-25T14:03:25.627785+0000","last_undegraded":"2024-04-25T14:03:25.627785+0000","last_fullsized":"2024-04-25T14:03:25.627785+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627453+0000","last_change":"2024-04-25T14:03:23.471660+0000","last_active":"2024-04-25T14:03:25.627453+0000","last_peered":"2024-04-25T14:03:25.627453+0000","last_clean":"2024-04-25T14:03:25.627453+0000","last_became_active":"2024-04-25T14:03:23.471344+0000","last_became_peered":"2024-04-25T14:03:23.471344+0000","last_unstale":"2024-04-25T14:03:25.627453+0000","last_undegraded":"2024-04-25T14:03:25.627453+0000","last_fullsized":"2024-04-25T14:03:25.627453+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:58.416724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472078+0000","last_change":"2024-04-25T14:03:22.465365+0000","last_active":"2024-04-25T14:03:24.472078+0000","last_peered":"2024-04-25T14:03:24.472078+0000","last_clean":"2024-04-25T14:03:24.472078+0000","last_became_active":"2024-04-25T14:03:22.465127+0000","last_became_peered":"2024-04-25T14:03:22.465127+0000","last_unstale":"2024-04-25T14:03:24.472078+0000","last_undegraded":"2024-04-25T14:03:24.472078+0000","last_fullsized":"2024-04-25T14:03:24.472078+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:34.447246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526471+0000","last_change":"2024-04-25T14:03:23.468884+0000","last_active":"2024-04-25T14:03:24.526471+0000","last_peered":"2024-04-25T14:03:24.526471+0000","last_clean":"2024-04-25T14:03:24.526471+0000","last_became_active":"2024-04-25T14:03:23.468584+0000","last_became_peered":"2024-04-25T14:03:23.468584+0000","last_unstale":"2024-04-25T14:03:24.526471+0000","last_undegraded":"2024-04-25T14:03:24.526471+0000","last_fullsized":"2024-04-25T14:03:24.526471+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:59.589691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473356+0000","last_change":"2024-04-25T14:03:22.467405+0000","last_active":"2024-04-25T14:03:24.473356+0000","last_peered":"2024-04-25T14:03:24.473356+0000","last_clean":"2024-04-25T14:03:24.473356+0000","last_became_active":"2024-04-25T14:03:22.467057+0000","last_became_peered":"2024-04-25T14:03:22.467057+0000","last_unstale":"2024-04-25T14:03:24.473356+0000","last_undegraded":"2024-04-25T14:03:24.473356+0000","last_fullsized":"2024-04-25T14:03:24.473356+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:03.552759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473530+0000","last_change":"2024-04-25T14:03:23.477620+0000","last_active":"2024-04-25T14:03:24.473530+0000","last_peered":"2024-04-25T14:03:24.473530+0000","last_clean":"2024-04-25T14:03:24.473530+0000","last_became_active":"2024-04-25T14:03:23.475898+0000","last_became_peered":"2024-04-25T14:03:23.475898+0000","last_unstale":"2024-04-25T14:03:24.473530+0000","last_undegraded":"2024-04-25T14:03:24.473530+0000","last_fullsized":"2024-04-25T14:03:24.473530+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:46:17.109982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471267+0000","last_change":"2024-04-25T14:03:22.456091+0000","last_active":"2024-04-25T14:03:24.471267+0000","last_peered":"2024-04-25T14:03:24.471267+0000","last_clean":"2024-04-25T14:03:24.471267+0000","last_became_active":"2024-04-25T14:03:22.455732+0000","last_became_peered":"2024-04-25T14:03:22.455732+0000","last_unstale":"2024-04-25T14:03:24.471267+0000","last_undegraded":"2024-04-25T14:03:24.471267+0000","last_fullsized":"2024-04-25T14:03:24.471267+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:10:34.130162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472654+0000","last_change":"2024-04-25T14:03:23.476806+0000","last_active":"2024-04-25T14:03:24.472654+0000","last_peered":"2024-04-25T14:03:24.472654+0000","last_clean":"2024-04-25T14:03:24.472654+0000","last_became_active":"2024-04-25T14:03:23.476485+0000","last_became_peered":"2024-04-25T14:03:23.476485+0000","last_unstale":"2024-04-25T14:03:24.472654+0000","last_undegraded":"2024-04-25T14:03:24.472654+0000","last_fullsized":"2024-04-25T14:03:24.472654+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627014+0000","last_change":"2024-04-25T14:03:22.464725+0000","last_active":"2024-04-25T14:03:25.627014+0000","last_peered":"2024-04-25T14:03:25.627014+0000","last_clean":"2024-04-25T14:03:25.627014+0000","last_became_active":"2024-04-25T14:03:22.464409+0000","last_became_peered":"2024-04-25T14:03:22.464409+0000","last_unstale":"2024-04-25T14:03:25.627014+0000","last_undegraded":"2024-04-25T14:03:25.627014+0000","last_fullsized":"2024-04-25T14:03:25.627014+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:40.729003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627908+0000","last_change":"2024-04-25T14:03:23.468975+0000","last_active":"2024-04-25T14:03:25.627908+0000","last_peered":"2024-04-25T14:03:25.627908+0000","last_clean":"2024-04-25T14:03:25.627908+0000","last_became_active":"2024-04-25T14:03:23.468661+0000","last_became_peered":"2024-04-25T14:03:23.468661+0000","last_unstale":"2024-04-25T14:03:25.627908+0000","last_undegraded":"2024-04-25T14:03:25.627908+0000","last_fullsized":"2024-04-25T14:03:25.627908+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:18.678250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471332+0000","last_change":"2024-04-25T14:03:22.453661+0000","last_active":"2024-04-25T14:03:24.471332+0000","last_peered":"2024-04-25T14:03:24.471332+0000","last_clean":"2024-04-25T14:03:24.471332+0000","last_became_active":"2024-04-25T14:03:22.453283+0000","last_became_peered":"2024-04-25T14:03:22.453283+0000","last_unstale":"2024-04-25T14:03:24.471332+0000","last_undegraded":"2024-04-25T14:03:24.471332+0000","last_fullsized":"2024-04-25T14:03:24.471332+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:11.817679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473811+0000","last_change":"2024-04-25T14:03:23.481586+0000","last_active":"2024-04-25T14:03:24.473811+0000","last_peered":"2024-04-25T14:03:24.473811+0000","last_clean":"2024-04-25T14:03:24.473811+0000","last_became_active":"2024-04-25T14:03:23.481287+0000","last_became_peered":"2024-04-25T14:03:23.481287+0000","last_unstale":"2024-04-25T14:03:24.473811+0000","last_undegraded":"2024-04-25T14:03:24.473811+0000","last_fullsized":"2024-04-25T14:03:24.473811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:20:41.087596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627675+0000","last_change":"2024-04-25T14:03:22.463623+0000","last_active":"2024-04-25T14:03:25.627675+0000","last_peered":"2024-04-25T14:03:25.627675+0000","last_clean":"2024-04-25T14:03:25.627675+0000","last_became_active":"2024-04-25T14:03:22.463289+0000","last_became_peered":"2024-04-25T14:03:22.463289+0000","last_unstale":"2024-04-25T14:03:25.627675+0000","last_undegraded":"2024-04-25T14:03:25.627675+0000","last_fullsized":"2024-04-25T14:03:25.627675+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:37.802148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471665+0000","last_change":"2024-04-25T14:03:23.469389+0000","last_active":"2024-04-25T14:03:24.471665+0000","last_peered":"2024-04-25T14:03:24.471665+0000","last_clean":"2024-04-25T14:03:24.471665+0000","last_became_active":"2024-04-25T14:03:23.469087+0000","last_became_peered":"2024-04-25T14:03:23.469087+0000","last_unstale":"2024-04-25T14:03:24.471665+0000","last_undegraded":"2024-04-25T14:03:24.471665+0000","last_fullsized":"2024-04-25T14:03:24.471665+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:19.446529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471556+0000","last_change":"2024-04-25T14:03:22.468120+0000","last_active":"2024-04-25T14:03:24.471556+0000","last_peered":"2024-04-25T14:03:24.471556+0000","last_clean":"2024-04-25T14:03:24.471556+0000","last_became_active":"2024-04-25T14:03:22.467909+0000","last_became_peered":"2024-04-25T14:03:22.467909+0000","last_unstale":"2024-04-25T14:03:24.471556+0000","last_undegraded":"2024-04-25T14:03:24.471556+0000","last_fullsized":"2024-04-25T14:03:24.471556+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:02.306058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526771+0000","last_change":"2024-04-25T14:03:23.465564+0000","last_active":"2024-04-25T14:03:24.526771+0000","last_peered":"2024-04-25T14:03:24.526771+0000","last_clean":"2024-04-25T14:03:24.526771+0000","last_became_active":"2024-04-25T14:03:23.465258+0000","last_became_peered":"2024-04-25T14:03:23.465258+0000","last_unstale":"2024-04-25T14:03:24.526771+0000","last_undegraded":"2024-04-25T14:03:24.526771+0000","last_fullsized":"2024-04-25T14:03:24.526771+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:58.295835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"37'3","reported_seq":17,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472585+0000","last_change":"2024-04-25T14:03:22.473042+0000","last_active":"2024-04-25T14:03:24.472585+0000","last_peered":"2024-04-25T14:03:24.472585+0000","last_clean":"2024-04-25T14:03:24.472585+0000","last_became_active":"2024-04-25T14:03:22.472772+0000","last_became_peered":"2024-04-25T14:03:22.472772+0000","last_unstale":"2024-04-25T14:03:24.472585+0000","last_undegraded":"2024-04-25T14:03:24.472585+0000","last_fullsized":"2024-04-25T14:03:24.472585+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:11.183503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473071+0000","last_change":"2024-04-25T14:03:23.478855+0000","last_active":"2024-04-25T14:03:24.473071+0000","last_peered":"2024-04-25T14:03:24.473071+0000","last_clean":"2024-04-25T14:03:24.473071+0000","last_became_active":"2024-04-25T14:03:23.478498+0000","last_became_peered":"2024-04-25T14:03:23.478498+0000","last_unstale":"2024-04-25T14:03:24.473071+0000","last_undegraded":"2024-04-25T14:03:24.473071+0000","last_fullsized":"2024-04-25T14:03:24.473071+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:09:50.950138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"37'6","reported_seq":20,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627684+0000","last_change":"2024-04-25T14:03:22.465174+0000","last_active":"2024-04-25T14:03:25.627684+0000","last_peered":"2024-04-25T14:03:25.627684+0000","last_clean":"2024-04-25T14:03:25.627684+0000","last_became_active":"2024-04-25T14:03:22.464901+0000","last_became_peered":"2024-04-25T14:03:22.464901+0000","last_unstale":"2024-04-25T14:03:25.627684+0000","last_undegraded":"2024-04-25T14:03:25.627684+0000","last_fullsized":"2024-04-25T14:03:25.627684+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:45:11.183503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526491+0000","last_change":"2024-04-25T14:03:23.475200+0000","last_active":"2024-04-25T14:03:24.526491+0000","last_peered":"2024-04-25T14:03:24.526491+0000","last_clean":"2024-04-25T14:03:24.526491+0000","last_became_active":"2024-04-25T14:03:23.474922+0000","last_became_peered":"2024-04-25T14:03:23.474922+0000","last_unstale":"2024-04-25T14:03:24.526491+0000","last_undegraded":"2024-04-25T14:03:24.526491+0000","last_fullsized":"2024-04-25T14:03:24.526491+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:11:03.870090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627877+0000","last_change":"2024-04-25T14:03:22.466901+0000","last_active":"2024-04-25T14:03:25.627877+0000","last_peered":"2024-04-25T14:03:25.627877+0000","last_clean":"2024-04-25T14:03:25.627877+0000","last_became_active":"2024-04-25T14:03:22.466600+0000","last_became_peered":"2024-04-25T14:03:22.466600+0000","last_unstale":"2024-04-25T14:03:25.627877+0000","last_undegraded":"2024-04-25T14:03:25.627877+0000","last_fullsized":"2024-04-25T14:03:25.627877+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:03.552759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627181+0000","last_change":"2024-04-25T14:03:23.473427+0000","last_active":"2024-04-25T14:03:25.627181+0000","last_peered":"2024-04-25T14:03:25.627181+0000","last_clean":"2024-04-25T14:03:25.627181+0000","last_became_active":"2024-04-25T14:03:23.473142+0000","last_became_peered":"2024-04-25T14:03:23.473142+0000","last_unstale":"2024-04-25T14:03:25.627181+0000","last_undegraded":"2024-04-25T14:03:25.627181+0000","last_fullsized":"2024-04-25T14:03:25.627181+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:30:39.336137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627294+0000","last_change":"2024-04-25T14:03:22.465444+0000","last_active":"2024-04-25T14:03:25.627294+0000","last_peered":"2024-04-25T14:03:25.627294+0000","last_clean":"2024-04-25T14:03:25.627294+0000","last_became_active":"2024-04-25T14:03:22.465099+0000","last_became_peered":"2024-04-25T14:03:22.465099+0000","last_unstale":"2024-04-25T14:03:25.627294+0000","last_undegraded":"2024-04-25T14:03:25.627294+0000","last_fullsized":"2024-04-25T14:03:25.627294+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:43:07.045482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471175+0000","last_change":"2024-04-25T14:03:23.468202+0000","last_active":"2024-04-25T14:03:24.471175+0000","last_peered":"2024-04-25T14:03:24.471175+0000","last_clean":"2024-04-25T14:03:24.471175+0000","last_became_active":"2024-04-25T14:03:23.467142+0000","last_became_peered":"2024-04-25T14:03:23.467142+0000","last_unstale":"2024-04-25T14:03:24.471175+0000","last_undegraded":"2024-04-25T14:03:24.471175+0000","last_fullsized":"2024-04-25T14:03:24.471175+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:23:58.295835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627522+0000","last_change":"2024-04-25T14:03:22.458534+0000","last_active":"2024-04-25T14:03:25.627522+0000","last_peered":"2024-04-25T14:03:25.627522+0000","last_clean":"2024-04-25T14:03:25.627522+0000","last_became_active":"2024-04-25T14:03:22.458229+0000","last_became_peered":"2024-04-25T14:03:22.458229+0000","last_unstale":"2024-04-25T14:03:25.627522+0000","last_undegraded":"2024-04-25T14:03:25.627522+0000","last_fullsized":"2024-04-25T14:03:25.627522+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:45:15.794799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627299+0000","last_change":"2024-04-25T14:03:23.468476+0000","last_active":"2024-04-25T14:03:25.627299+0000","last_peered":"2024-04-25T14:03:25.627299+0000","last_clean":"2024-04-25T14:03:25.627299+0000","last_became_active":"2024-04-25T14:03:23.467830+0000","last_became_peered":"2024-04-25T14:03:23.467830+0000","last_unstale":"2024-04-25T14:03:25.627299+0000","last_undegraded":"2024-04-25T14:03:25.627299+0000","last_fullsized":"2024-04-25T14:03:25.627299+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:21.063720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"37'8","reported_seq":22,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471170+0000","last_change":"2024-04-25T14:03:22.457933+0000","last_active":"2024-04-25T14:03:24.471170+0000","last_peered":"2024-04-25T14:03:24.471170+0000","last_clean":"2024-04-25T14:03:24.471170+0000","last_became_active":"2024-04-25T14:03:22.457548+0000","last_became_peered":"2024-04-25T14:03:22.457548+0000","last_unstale":"2024-04-25T14:03:24.471170+0000","last_undegraded":"2024-04-25T14:03:24.471170+0000","last_fullsized":"2024-04-25T14:03:24.471170+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627641+0000","last_change":"2024-04-25T14:03:23.472401+0000","last_active":"2024-04-25T14:03:25.627641+0000","last_peered":"2024-04-25T14:03:25.627641+0000","last_clean":"2024-04-25T14:03:25.627641+0000","last_became_active":"2024-04-25T14:03:23.472116+0000","last_became_peered":"2024-04-25T14:03:23.472116+0000","last_unstale":"2024-04-25T14:03:25.627641+0000","last_undegraded":"2024-04-25T14:03:25.627641+0000","last_fullsized":"2024-04-25T14:03:25.627641+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:48:17.712130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473428+0000","last_change":"2024-04-25T14:03:22.470135+0000","last_active":"2024-04-25T14:03:24.473428+0000","last_peered":"2024-04-25T14:03:24.473428+0000","last_clean":"2024-04-25T14:03:24.473428+0000","last_became_active":"2024-04-25T14:03:22.469798+0000","last_became_peered":"2024-04-25T14:03:22.469798+0000","last_unstale":"2024-04-25T14:03:24.473428+0000","last_undegraded":"2024-04-25T14:03:24.473428+0000","last_fullsized":"2024-04-25T14:03:24.473428+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:43.711638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472452+0000","last_change":"2024-04-25T14:03:23.473341+0000","last_active":"2024-04-25T14:03:24.472452+0000","last_peered":"2024-04-25T14:03:24.472452+0000","last_clean":"2024-04-25T14:03:24.472452+0000","last_became_active":"2024-04-25T14:03:23.473018+0000","last_became_peered":"2024-04-25T14:03:23.473018+0000","last_unstale":"2024-04-25T14:03:24.472452+0000","last_undegraded":"2024-04-25T14:03:24.472452+0000","last_fullsized":"2024-04-25T14:03:24.472452+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472690+0000","last_change":"2024-04-25T14:03:22.462735+0000","last_active":"2024-04-25T14:03:24.472690+0000","last_peered":"2024-04-25T14:03:24.472690+0000","last_clean":"2024-04-25T14:03:24.472690+0000","last_became_active":"2024-04-25T14:03:22.462427+0000","last_became_peered":"2024-04-25T14:03:22.462427+0000","last_unstale":"2024-04-25T14:03:24.472690+0000","last_undegraded":"2024-04-25T14:03:24.472690+0000","last_fullsized":"2024-04-25T14:03:24.472690+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:07:40.729003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472525+0000","last_change":"2024-04-25T14:03:23.472230+0000","last_active":"2024-04-25T14:03:24.472525+0000","last_peered":"2024-04-25T14:03:24.472525+0000","last_clean":"2024-04-25T14:03:24.472525+0000","last_became_active":"2024-04-25T14:03:23.471916+0000","last_became_peered":"2024-04-25T14:03:23.471916+0000","last_unstale":"2024-04-25T14:03:24.472525+0000","last_undegraded":"2024-04-25T14:03:24.472525+0000","last_fullsized":"2024-04-25T14:03:24.472525+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:44.718503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"37'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473747+0000","last_change":"2024-04-25T14:03:22.457434+0000","last_active":"2024-04-25T14:03:24.473747+0000","last_peered":"2024-04-25T14:03:24.473747+0000","last_clean":"2024-04-25T14:03:24.473747+0000","last_became_active":"2024-04-25T14:03:22.457122+0000","last_became_peered":"2024-04-25T14:03:22.457122+0000","last_unstale":"2024-04-25T14:03:24.473747+0000","last_undegraded":"2024-04-25T14:03:24.473747+0000","last_fullsized":"2024-04-25T14:03:24.473747+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:16:15.859362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473865+0000","last_change":"2024-04-25T14:03:23.470502+0000","last_active":"2024-04-25T14:03:24.473865+0000","last_peered":"2024-04-25T14:03:24.473865+0000","last_clean":"2024-04-25T14:03:24.473865+0000","last_became_active":"2024-04-25T14:03:23.470197+0000","last_became_peered":"2024-04-25T14:03:23.470197+0000","last_unstale":"2024-04-25T14:03:24.473865+0000","last_undegraded":"2024-04-25T14:03:24.473865+0000","last_fullsized":"2024-04-25T14:03:24.473865+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:35.454111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627397+0000","last_change":"2024-04-25T14:03:22.459335+0000","last_active":"2024-04-25T14:03:25.627397+0000","last_peered":"2024-04-25T14:03:25.627397+0000","last_clean":"2024-04-25T14:03:25.627397+0000","last_became_active":"2024-04-25T14:03:22.459008+0000","last_became_peered":"2024-04-25T14:03:22.459008+0000","last_unstale":"2024-04-25T14:03:25.627397+0000","last_undegraded":"2024-04-25T14:03:25.627397+0000","last_fullsized":"2024-04-25T14:03:25.627397+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:39:40.496688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471399+0000","last_change":"2024-04-25T14:03:23.467432+0000","last_active":"2024-04-25T14:03:24.471399+0000","last_peered":"2024-04-25T14:03:24.471399+0000","last_clean":"2024-04-25T14:03:24.471399+0000","last_became_active":"2024-04-25T14:03:23.467100+0000","last_became_peered":"2024-04-25T14:03:23.467100+0000","last_unstale":"2024-04-25T14:03:24.471399+0000","last_undegraded":"2024-04-25T14:03:24.471399+0000","last_fullsized":"2024-04-25T14:03:24.471399+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:03.945722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473464+0000","last_change":"2024-04-25T14:03:22.463630+0000","last_active":"2024-04-25T14:03:24.473464+0000","last_peered":"2024-04-25T14:03:24.473464+0000","last_clean":"2024-04-25T14:03:24.473464+0000","last_became_active":"2024-04-25T14:03:22.463311+0000","last_became_peered":"2024-04-25T14:03:22.463311+0000","last_unstale":"2024-04-25T14:03:24.473464+0000","last_undegraded":"2024-04-25T14:03:24.473464+0000","last_fullsized":"2024-04-25T14:03:24.473464+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:39:50.682001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526551+0000","last_change":"2024-04-25T14:03:23.470162+0000","last_active":"2024-04-25T14:03:24.526551+0000","last_peered":"2024-04-25T14:03:24.526551+0000","last_clean":"2024-04-25T14:03:24.526551+0000","last_became_active":"2024-04-25T14:03:23.469900+0000","last_became_peered":"2024-04-25T14:03:23.469900+0000","last_unstale":"2024-04-25T14:03:24.526551+0000","last_undegraded":"2024-04-25T14:03:24.526551+0000","last_fullsized":"2024-04-25T14:03:24.526551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:43:58.837156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473657+0000","last_change":"2024-04-25T14:03:22.455477+0000","last_active":"2024-04-25T14:03:24.473657+0000","last_peered":"2024-04-25T14:03:24.473657+0000","last_clean":"2024-04-25T14:03:24.473657+0000","last_became_active":"2024-04-25T14:03:22.455138+0000","last_became_peered":"2024-04-25T14:03:22.455138+0000","last_unstale":"2024-04-25T14:03:24.473657+0000","last_undegraded":"2024-04-25T14:03:24.473657+0000","last_fullsized":"2024-04-25T14:03:24.473657+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:04:34.353549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473226+0000","last_change":"2024-04-25T14:03:23.478492+0000","last_active":"2024-04-25T14:03:24.473226+0000","last_peered":"2024-04-25T14:03:24.473226+0000","last_clean":"2024-04-25T14:03:24.473226+0000","last_became_active":"2024-04-25T14:03:23.478120+0000","last_became_peered":"2024-04-25T14:03:23.478120+0000","last_unstale":"2024-04-25T14:03:24.473226+0000","last_undegraded":"2024-04-25T14:03:24.473226+0000","last_fullsized":"2024-04-25T14:03:24.473226+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:43:58.416724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472752+0000","last_change":"2024-04-25T14:03:22.459674+0000","last_active":"2024-04-25T14:03:24.472752+0000","last_peered":"2024-04-25T14:03:24.472752+0000","last_clean":"2024-04-25T14:03:24.472752+0000","last_became_active":"2024-04-25T14:03:22.459373+0000","last_became_peered":"2024-04-25T14:03:22.459373+0000","last_unstale":"2024-04-25T14:03:24.472752+0000","last_undegraded":"2024-04-25T14:03:24.472752+0000","last_fullsized":"2024-04-25T14:03:24.472752+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:45:47.626684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627821+0000","last_change":"2024-04-25T14:03:23.468724+0000","last_active":"2024-04-25T14:03:25.627821+0000","last_peered":"2024-04-25T14:03:25.627821+0000","last_clean":"2024-04-25T14:03:25.627821+0000","last_became_active":"2024-04-25T14:03:23.468222+0000","last_became_peered":"2024-04-25T14:03:23.468222+0000","last_unstale":"2024-04-25T14:03:25.627821+0000","last_undegraded":"2024-04-25T14:03:25.627821+0000","last_fullsized":"2024-04-25T14:03:25.627821+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:46:25.322719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473802+0000","last_change":"2024-04-25T14:03:22.462280+0000","last_active":"2024-04-25T14:03:24.473802+0000","last_peered":"2024-04-25T14:03:24.473802+0000","last_clean":"2024-04-25T14:03:24.473802+0000","last_became_active":"2024-04-25T14:03:22.461941+0000","last_became_peered":"2024-04-25T14:03:22.461941+0000","last_unstale":"2024-04-25T14:03:24.473802+0000","last_undegraded":"2024-04-25T14:03:24.473802+0000","last_fullsized":"2024-04-25T14:03:24.473802+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:18:08.832919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627436+0000","last_change":"2024-04-25T14:03:23.478576+0000","last_active":"2024-04-25T14:03:25.627436+0000","last_peered":"2024-04-25T14:03:25.627436+0000","last_clean":"2024-04-25T14:03:25.627436+0000","last_became_active":"2024-04-25T14:03:23.478257+0000","last_became_peered":"2024-04-25T14:03:23.478257+0000","last_unstale":"2024-04-25T14:03:25.627436+0000","last_undegraded":"2024-04-25T14:03:25.627436+0000","last_fullsized":"2024-04-25T14:03:25.627436+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:15:30.179527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"37'2","reported_seq":16,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473547+0000","last_change":"2024-04-25T14:03:22.472916+0000","last_active":"2024-04-25T14:03:24.473547+0000","last_peered":"2024-04-25T14:03:24.473547+0000","last_clean":"2024-04-25T14:03:24.473547+0000","last_became_active":"2024-04-25T14:03:22.472691+0000","last_became_peered":"2024-04-25T14:03:22.472691+0000","last_unstale":"2024-04-25T14:03:24.473547+0000","last_undegraded":"2024-04-25T14:03:24.473547+0000","last_fullsized":"2024-04-25T14:03:24.473547+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:10:37.802148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471839+0000","last_change":"2024-04-25T14:03:23.467891+0000","last_active":"2024-04-25T14:03:24.471839+0000","last_peered":"2024-04-25T14:03:24.471839+0000","last_clean":"2024-04-25T14:03:24.471839+0000","last_became_active":"2024-04-25T14:03:23.467578+0000","last_became_peered":"2024-04-25T14:03:23.467578+0000","last_unstale":"2024-04-25T14:03:24.471839+0000","last_undegraded":"2024-04-25T14:03:24.471839+0000","last_fullsized":"2024-04-25T14:03:24.471839+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:06:28.888914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"38'1","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473870+0000","last_change":"2024-04-25T14:03:22.466332+0000","last_active":"2024-04-25T14:03:24.473870+0000","last_peered":"2024-04-25T14:03:24.473870+0000","last_clean":"2024-04-25T14:03:24.473870+0000","last_became_active":"2024-04-25T14:03:22.466032+0000","last_became_peered":"2024-04-25T14:03:22.466032+0000","last_unstale":"2024-04-25T14:03:24.473870+0000","last_undegraded":"2024-04-25T14:03:24.473870+0000","last_fullsized":"2024-04-25T14:03:24.473870+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:31:50.091226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627592+0000","last_change":"2024-04-25T14:03:23.475055+0000","last_active":"2024-04-25T14:03:25.627592+0000","last_peered":"2024-04-25T14:03:25.627592+0000","last_clean":"2024-04-25T14:03:25.627592+0000","last_became_active":"2024-04-25T14:03:23.474733+0000","last_became_peered":"2024-04-25T14:03:23.474733+0000","last_unstale":"2024-04-25T14:03:25.627592+0000","last_undegraded":"2024-04-25T14:03:25.627592+0000","last_fullsized":"2024-04-25T14:03:25.627592+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:06:39.710834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526620+0000","last_change":"2024-04-25T14:03:22.467520+0000","last_active":"2024-04-25T14:03:24.526620+0000","last_peered":"2024-04-25T14:03:24.526620+0000","last_clean":"2024-04-25T14:03:24.526620+0000","last_became_active":"2024-04-25T14:03:22.467223+0000","last_became_peered":"2024-04-25T14:03:22.467223+0000","last_unstale":"2024-04-25T14:03:24.526620+0000","last_undegraded":"2024-04-25T14:03:24.526620+0000","last_fullsized":"2024-04-25T14:03:24.526620+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:20.831468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526775+0000","last_change":"2024-04-25T14:03:23.466245+0000","last_active":"2024-04-25T14:03:24.526775+0000","last_peered":"2024-04-25T14:03:24.526775+0000","last_clean":"2024-04-25T14:03:24.526775+0000","last_became_active":"2024-04-25T14:03:23.465975+0000","last_became_peered":"2024-04-25T14:03:23.465975+0000","last_unstale":"2024-04-25T14:03:24.526775+0000","last_undegraded":"2024-04-25T14:03:24.526775+0000","last_fullsized":"2024-04-25T14:03:24.526775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:42:03.945722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.526745+0000","last_change":"2024-04-25T14:03:22.467232+0000","last_active":"2024-04-25T14:03:24.526745+0000","last_peered":"2024-04-25T14:03:24.526745+0000","last_clean":"2024-04-25T14:03:24.526745+0000","last_became_active":"2024-04-25T14:03:22.466908+0000","last_became_peered":"2024-04-25T14:03:22.466908+0000","last_unstale":"2024-04-25T14:03:24.526745+0000","last_undegraded":"2024-04-25T14:03:24.526745+0000","last_fullsized":"2024-04-25T14:03:24.526745+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.764344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627113+0000","last_change":"2024-04-25T14:03:23.468220+0000","last_active":"2024-04-25T14:03:25.627113+0000","last_peered":"2024-04-25T14:03:25.627113+0000","last_clean":"2024-04-25T14:03:25.627113+0000","last_became_active":"2024-04-25T14:03:23.467815+0000","last_became_peered":"2024-04-25T14:03:23.467815+0000","last_unstale":"2024-04-25T14:03:25.627113+0000","last_undegraded":"2024-04-25T14:03:25.627113+0000","last_fullsized":"2024-04-25T14:03:25.627113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:54:04.559624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627705+0000","last_change":"2024-04-25T14:03:22.460680+0000","last_active":"2024-04-25T14:03:25.627705+0000","last_peered":"2024-04-25T14:03:25.627705+0000","last_clean":"2024-04-25T14:03:25.627705+0000","last_became_active":"2024-04-25T14:03:22.460354+0000","last_became_peered":"2024-04-25T14:03:22.460354+0000","last_unstale":"2024-04-25T14:03:25.627705+0000","last_undegraded":"2024-04-25T14:03:25.627705+0000","last_fullsized":"2024-04-25T14:03:25.627705+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:06:11.817679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471885+0000","last_change":"2024-04-25T14:03:23.478976+0000","last_active":"2024-04-25T14:03:24.471885+0000","last_peered":"2024-04-25T14:03:24.471885+0000","last_clean":"2024-04-25T14:03:24.471885+0000","last_became_active":"2024-04-25T14:03:23.478509+0000","last_became_peered":"2024-04-25T14:03:23.478509+0000","last_unstale":"2024-04-25T14:03:24.471885+0000","last_undegraded":"2024-04-25T14:03:24.471885+0000","last_fullsized":"2024-04-25T14:03:24.471885+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:47:03.312923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.470832+0000","last_change":"2024-04-25T14:03:22.465180+0000","last_active":"2024-04-25T14:03:24.470832+0000","last_peered":"2024-04-25T14:03:24.470832+0000","last_clean":"2024-04-25T14:03:24.470832+0000","last_became_active":"2024-04-25T14:03:22.464890+0000","last_became_peered":"2024-04-25T14:03:22.464890+0000","last_unstale":"2024-04-25T14:03:24.470832+0000","last_undegraded":"2024-04-25T14:03:24.470832+0000","last_fullsized":"2024-04-25T14:03:24.470832+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:16:40.514183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473200+0000","last_change":"2024-04-25T14:03:23.468642+0000","last_active":"2024-04-25T14:03:24.473200+0000","last_peered":"2024-04-25T14:03:24.473200+0000","last_clean":"2024-04-25T14:03:24.473200+0000","last_became_active":"2024-04-25T14:03:23.467883+0000","last_became_peered":"2024-04-25T14:03:23.467883+0000","last_unstale":"2024-04-25T14:03:24.473200+0000","last_undegraded":"2024-04-25T14:03:24.473200+0000","last_fullsized":"2024-04-25T14:03:24.473200+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:45:46.601549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:25.627187+0000","last_change":"2024-04-25T14:03:22.463281+0000","last_active":"2024-04-25T14:03:25.627187+0000","last_peered":"2024-04-25T14:03:25.627187+0000","last_clean":"2024-04-25T14:03:25.627187+0000","last_became_active":"2024-04-25T14:03:22.462977+0000","last_became_peered":"2024-04-25T14:03:22.462977+0000","last_unstale":"2024-04-25T14:03:25.627187+0000","last_undegraded":"2024-04-25T14:03:25.627187+0000","last_fullsized":"2024-04-25T14:03:25.627187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:40:43.711638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.473469+0000","last_change":"2024-04-25T14:03:23.476670+0000","last_active":"2024-04-25T14:03:24.473469+0000","last_peered":"2024-04-25T14:03:24.473469+0000","last_clean":"2024-04-25T14:03:24.473469+0000","last_became_active":"2024-04-25T14:03:23.476307+0000","last_became_peered":"2024-04-25T14:03:23.476307+0000","last_unstale":"2024-04-25T14:03:24.473469+0000","last_undegraded":"2024-04-25T14:03:24.473469+0000","last_fullsized":"2024-04-25T14:03:24.473469+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:22.440461+0000","last_clean_scrub_stamp":"2024-04-25T14:03:22.440461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:43:18.678250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.472168+0000","last_change":"2024-04-25T14:03:22.463970+0000","last_active":"2024-04-25T14:03:24.472168+0000","last_peered":"2024-04-25T14:03:24.472168+0000","last_clean":"2024-04-25T14:03:24.472168+0000","last_became_active":"2024-04-25T14:03:22.463642+0000","last_became_peered":"2024-04-25T14:03:22.463642+0000","last_unstale":"2024-04-25T14:03:24.472168+0000","last_undegraded":"2024-04-25T14:03:24.472168+0000","last_fullsized":"2024-04-25T14:03:24.472168+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":35,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:03:21.433596+0000","last_clean_scrub_stamp":"2024-04-25T14:03:21.433596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:28:39.764344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":146,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-25T14:03:24.471460+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:03:24.471460+0000","last_peered":"2024-04-25T14:03:24.471460+0000","last_clean":"2024-04-25T14:03:24.471460+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:03:24.471460+0000","last_undegraded":"2024-04-25T14:03:24.471460+0000","last_fullsized":"2024-04-25T14:03:24.471460+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":782086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574872,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":190148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":190640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27464,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344376,"statfs":{"total":96636764160,"available":96608641024,"internally_reserved":0,"allocated":454656,"data_stored":191682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":190126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343804,"statfs":{"total":96636764160,"available":96608055296,"internally_reserved":0,"allocated":1036288,"data_stored":780438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:03:32.259 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T14:03:32.259 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T14:03:32.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.410+0000 7f860a0b1640 1 Processor -- start 2024-04-25T14:03:32.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.410+0000 7f860a0b1640 1 -- start start 2024-04-25T14:03:32.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f860415f2b0 0x7f8604072ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f86040b2f20 0x7f86040733e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 0x7f8604162000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f860415ee20 con 0x7f860415f2b0 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f860415e7f0 con 0x7f86040b2f20 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f860a0b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8604162920 con 0x7f8604161bf0 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f860415f2b0 0x7f8604072ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f860415f2b0 0x7f8604072ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45188/0 (socket says 172.21.15.150:45188) 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 -- 172.21.15.150:0/3993330360 learned_addr learned my addr 172.21.15.150:0/3993330360 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:32.413 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f8602ffd640 1 --2- 172.21.15.150:0/3993330360 >> v2:172.21.15.187:3300/0 conn(0x7f86040b2f20 0x7f86040733e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f8603fff640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 0x7f8604162000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 -- 172.21.15.150:0/3993330360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 msgr2=0x7f8604162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 0x7f8604162000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 -- 172.21.15.150:0/3993330360 >> v2:172.21.15.187:3300/0 conn(0x7f86040b2f20 msgr2=0x7f86040733e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 --2- 172.21.15.150:0/3993330360 >> v2:172.21.15.187:3300/0 conn(0x7f86040b2f20 0x7f86040733e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f86037fe640 1 -- 172.21.15.150:0/3993330360 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860415eab0 con 0x7f860415f2b0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.412+0000 7f8603fff640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 0x7f8604162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.413+0000 7f86037fe640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f8604072ea0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86040d1510 tx=0x7f85f402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f28521d43c9ffb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.413+0000 7f86027fc640 1 -- 172.21.15.150:0/3993330360 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4037070 con 0x7f860415f2b0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.413+0000 7f86027fc640 1 -- 172.21.15.150:0/3993330360 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f4032440 con 0x7f860415f2b0 2024-04-25T14:03:32.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.413+0000 7f86027fc640 1 -- 172.21.15.150:0/3993330360 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f4032810 con 0x7f860415f2b0 2024-04-25T14:03:32.415 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 -- 172.21.15.150:0/3993330360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 msgr2=0x7f8604072ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f8604072ea0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f86040d1510 tx=0x7f85f402a140 comp rx=0 tx=0).stop 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 -- 172.21.15.150:0/3993330360 shutdown_connections 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8604161bf0 0x7f8604162000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 --2- 172.21.15.150:0/3993330360 >> v2:172.21.15.187:3300/0 conn(0x7f86040b2f20 0x7f86040733e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 --2- 172.21.15.150:0/3993330360 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f8604072ea0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 -- 172.21.15.150:0/3993330360 >> 172.21.15.150:0/3993330360 conn(0x7f86040d39b0 msgr2=0x7f86040d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 -- 172.21.15.150:0/3993330360 shutdown_connections 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.414+0000 7f860a0b1640 1 -- 172.21.15.150:0/3993330360 wait complete. 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.416+0000 7f860a0b1640 1 Processor -- start 2024-04-25T14:03:32.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.416+0000 7f860a0b1640 1 -- start start 2024-04-25T14:03:32.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86040b2f20 0x7f86041f7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.418 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 0x7f86040c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86040702a0 con 0x7f860415f2b0 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8604070120 con 0x7f8604161bf0 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f860a0b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8604070420 con 0x7f86040b2f20 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45198/0 (socket says 172.21.15.150:45198) 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 -- 172.21.15.150:0/545816153 learned_addr learned my addr 172.21.15.150:0/545816153 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f86037fe640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86040b2f20 0x7f86041f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8603fff640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 0x7f86040c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 -- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86040b2f20 msgr2=0x7f86041f7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86040b2f20 0x7f86041f7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.417+0000 7f8602ffd640 1 -- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 msgr2=0x7f86040c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f8602ffd640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 0x7f86040c8ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f8602ffd640 1 -- 172.21.15.150:0/545816153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f860414ce10 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f8603fff640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 0x7f86040c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f8602ffd640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85f800d830 tx=0x7f85f800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f8018030 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85f800be10 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85f80098b0 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f860414d0f0 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.418+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f860414ba50 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.419+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8604076a40 con 0x7f860415f2b0 2024-04-25T14:03:32.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.419+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f85f801b070 con 0x7f860415f2b0 2024-04-25T14:03:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.420+0000 7f85ebfff640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 0x7f85f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.420+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f85f80955b0 con 0x7f860415f2b0 2024-04-25T14:03:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.420+0000 7f86037fe640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 0x7f85f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.421 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.420+0000 7f86037fe640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 0x7f85f00769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85f4034000 tx=0x7f85f402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.425+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f85f805fbb0 con 0x7f860415f2b0 2024-04-25T14:03:32.668 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.668+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f86040d1e00 con 0x7f860415f2b0 2024-04-25T14:03:32.670 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.671+0000 7f85ebfff640 1 -- 172.21.15.150:0/545816153 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f85f805f4b0 con 0x7f860415f2b0 2024-04-25T14:03:32.670 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-04-25T14:03:32.670 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ== 2024-04-25T14:03:32.671 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-04-25T14:03:32.671 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-04-25T14:03:32.671 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T14:03:32.671 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-04-25T14:03:32.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 msgr2=0x7f85f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.672 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 0x7f85f00769f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f85f4034000 tx=0x7f85f402e000 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 msgr2=0x7f86040cd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f85f800d830 tx=0x7f85f800dcf0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 shutdown_connections 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f85f00744e0 0x7f85f00769f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8604161bf0 0x7f86040c8ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f860415f2b0 0x7f86040cd770 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 --2- 172.21.15.150:0/545816153 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86040b2f20 0x7f86041f7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 >> 172.21.15.150:0/545816153 conn(0x7f86040d39b0 msgr2=0x7f86040b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 shutdown_connections 2024-04-25T14:03:32.673 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.673+0000 7f860a0b1640 1 -- 172.21.15.150:0/545816153 wait complete. 2024-04-25T14:03:32.683 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:32.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.792+0000 7f5109ba8640 1 Processor -- start 2024-04-25T14:03:32.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.792+0000 7f5109ba8640 1 -- start start 2024-04-25T14:03:32.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51040d1120 0x7f5104070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5104161b10 0x7f5104161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5104154950 0x7f5104154d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f51040d1530 con 0x7f5104154950 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5104162460 con 0x7f5104161b10 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5109ba8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f51040aec30 con 0x7f51040d1120 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5104154950 0x7f5104154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f5104154950 0x7f5104154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45208/0 (socket says 172.21.15.150:45208) 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 -- 172.21.15.150:0/2191480727 learned_addr learned my addr 172.21.15.150:0/2191480727 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5102ffd640 1 --2- 172.21.15.150:0/2191480727 >> v2:172.21.15.187:3300/0 conn(0x7f5104161b10 0x7f5104161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f51037fe640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51040d1120 0x7f5104070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 -- 172.21.15.150:0/2191480727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51040d1120 msgr2=0x7f5104070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51040d1120 0x7f5104070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 -- 172.21.15.150:0/2191480727 >> v2:172.21.15.187:3300/0 conn(0x7f5104161b10 msgr2=0x7f5104161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:32.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 --2- 172.21.15.150:0/2191480727 >> v2:172.21.15.187:3300/0 conn(0x7f5104161b10 0x7f5104161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.794+0000 7f5103fff640 1 -- 172.21.15.150:0/2191480727 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51040aef10 con 0x7f5104154950 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.795+0000 7f5103fff640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f5104154d60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f51040744e0 tx=0x7f50fc02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e2b152804531de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.795+0000 7f51027fc640 1 -- 172.21.15.150:0/2191480727 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc036070 con 0x7f5104154950 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.795+0000 7f51027fc640 1 -- 172.21.15.150:0/2191480727 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50fc031600 con 0x7f5104154950 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.795+0000 7f51027fc640 1 -- 172.21.15.150:0/2191480727 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50fc0319d0 con 0x7f5104154950 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 -- 172.21.15.150:0/2191480727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 msgr2=0x7f5104154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f5104154d60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f51040744e0 tx=0x7f50fc02a570 comp rx=0 tx=0).stop 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 -- 172.21.15.150:0/2191480727 shutdown_connections 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f5104154d60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 --2- 172.21.15.150:0/2191480727 >> v2:172.21.15.187:3300/0 conn(0x7f5104161b10 0x7f5104161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 --2- 172.21.15.150:0/2191480727 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f51040d1120 0x7f5104070ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 -- 172.21.15.150:0/2191480727 >> 172.21.15.150:0/2191480727 conn(0x7f51040d1ae0 msgr2=0x7f51040d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:32.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 -- 172.21.15.150:0/2191480727 shutdown_connections 2024-04-25T14:03:32.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.796+0000 7f5109ba8640 1 -- 172.21.15.150:0/2191480727 wait complete. 2024-04-25T14:03:32.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.797+0000 7f5109ba8640 1 Processor -- start 2024-04-25T14:03:32.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.798+0000 7f5109ba8640 1 -- start start 2024-04-25T14:03:32.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51040d1120 0x7f51041ea090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5104161b10 0x7f51040cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f51041552a0 con 0x7f5104154950 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f51040af4d0 con 0x7f51040d1120 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5109ba8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f510413d690 con 0x7f5104161b10 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45224/0 (socket says 172.21.15.150:45224) 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 -- 172.21.15.150:0/3215603603 learned_addr learned my addr 172.21.15.150:0/3215603603 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5103fff640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5104161b10 0x7f51040cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f51037fe640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51040d1120 0x7f51041ea090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 -- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5104161b10 msgr2=0x7f51040cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5104161b10 0x7f51040cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 -- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51040d1120 msgr2=0x7f51041ea090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:32.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51040d1120 0x7f51041ea090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.799+0000 7f5102ffd640 1 -- 172.21.15.150:0/3215603603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f510413bb90 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f5102ffd640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f50f800e870 tx=0x7f50f800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8017070 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f800ee60 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f510413e4b0 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f800cb80 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.800+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f510413be30 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.801+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5104076a40 con 0x7f5104154950 2024-04-25T14:03:32.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.802+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f50f801a070 con 0x7f5104154950 2024-04-25T14:03:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.802+0000 7f50ebfff640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 0x7f50f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:32.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.802+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f50f8095170 con 0x7f5104154950 2024-04-25T14:03:32.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.802+0000 7f51037fe640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 0x7f50f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:32.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.802+0000 7f51037fe640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 0x7f50f00769f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50f40099b0 tx=0x7f50f4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:32.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:32.807+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f50f805f800 con 0x7f5104154950 2024-04-25T14:03:33.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.017+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5104070250 con 0x7f5104154950 2024-04-25T14:03:33.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.018+0000 7f50ebfff640 1 -- 172.21.15.150:0/3215603603 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5184 (secure 0 0 0) 0x7f50f805f100 con 0x7f5104154950 2024-04-25T14:03:33.018 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:33.018 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/21568960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":21568960},{"type":"v1","addr":"172.21.15.187:6834","nonce":21568960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5176,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/265568101","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":265568101},{"type":"v1","addr":"172.21.15.150:6837","nonce":265568101}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5185,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1241446695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":1241446695},{"type":"v1","addr":"172.21.15.187:6835","nonce":1241446695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:25.572204+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5349},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5349":{"gid":5349,"name":"c","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.150:6836/3196344346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3196344346},{"type":"v1","addr":"172.21.15.150:6836","nonce":3196344346}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5349,"qdb_cluster":[5349]},"id":2}]} 2024-04-25T14:03:33.020 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 16 2024-04-25T14:03:33.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.019+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 msgr2=0x7f50f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.019+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 0x7f50f00769f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f50f40099b0 tx=0x7f50f4008040 comp rx=0 tx=0).stop 2024-04-25T14:03:33.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 msgr2=0x7f510413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f50f800e870 tx=0x7f50f800ed30 comp rx=0 tx=0).stop 2024-04-25T14:03:33.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 shutdown_connections 2024-04-25T14:03:33.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50f00744e0 0x7f50f00769f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5104161b10 0x7f51040cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5104154950 0x7f510413db50 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 --2- 172.21.15.150:0/3215603603 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f51040d1120 0x7f51041ea090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 >> 172.21.15.150:0/3215603603 conn(0x7f51040d1ae0 msgr2=0x7f51040d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 shutdown_connections 2024-04-25T14:03:33.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.020+0000 7f5109ba8640 1 -- 172.21.15.150:0/3215603603 wait complete. 2024-04-25T14:03:33.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:03:22.813096+0000', 'modified': '2024-04-25T14:03:25.572204+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5349}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5349': {'gid': 5349, 'name': 'c', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.150:6836/3196344346', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 3196344346}, {'type': 'v1', 'addr': '172.21.15.150:6836', 'nonce': 3196344346}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5349, 'qdb_cluster': [5349]} 2024-04-25T14:03:33.434 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:03:33.434 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T14:03:33.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.545+0000 7f557d414640 1 Processor -- start 2024-04-25T14:03:33.545 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.545+0000 7f557d414640 1 -- start start 2024-04-25T14:03:33.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f557d414640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55780b4c10 0x7f5578160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f557d414640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55780b91f0 0x7f557806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f557d414640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55780b9600 con 0x7f557806d630 2024-04-25T14:03:33.549 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f557d414640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f557806cc60 con 0x7f55780b91f0 2024-04-25T14:03:33.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f557d414640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55780d4f20 con 0x7f55780b4c10 2024-04-25T14:03:33.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f55767fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55780b91f0 0x7f557806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f55767fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f55780b91f0 0x7f557806d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45236/0 (socket says 172.21.15.150:45236) 2024-04-25T14:03:33.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.546+0000 7f55767fc640 1 -- 172.21.15.150:0/3558870661 learned_addr learned my addr 172.21.15.150:0/3558870661 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f5576ffd640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55780b4c10 0x7f5578160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f55767fc640 1 -- 172.21.15.150:0/3558870661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55780b4c10 msgr2=0x7f5578160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f55767fc640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55780b4c10 0x7f5578160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f55767fc640 1 -- 172.21.15.150:0/3558870661 >> v1:172.21.15.187:6789/0 conn(0x7f557806d630 legacy=0x7f5578163c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.551 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f55767fc640 1 -- 172.21.15.150:0/3558870661 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55780d5200 con 0x7f55780b91f0 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.547+0000 7f55767fc640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b91f0 0x7f557806d0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f557806cde0 tx=0x7f556c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a9cf541e31b761 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f5575ffb640 1 -- 172.21.15.150:0/3558870661 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c037070 con 0x7f55780b91f0 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f5575ffb640 1 -- 172.21.15.150:0/3558870661 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c032440 con 0x7f55780b91f0 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f5575ffb640 1 -- 172.21.15.150:0/3558870661 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c032810 con 0x7f55780b91f0 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 -- 172.21.15.150:0/3558870661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b91f0 msgr2=0x7f557806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b91f0 0x7f557806d0f0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f557806cde0 tx=0x7f556c02a250 comp rx=0 tx=0).stop 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 -- 172.21.15.150:0/3558870661 shutdown_connections 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b91f0 0x7f557806d0f0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 --2- 172.21.15.150:0/3558870661 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f55780b4c10 0x7f5578160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 -- 172.21.15.150:0/3558870661 >> 172.21.15.150:0/3558870661 conn(0x7f55780b7690 msgr2=0x7f55780b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 -- 172.21.15.150:0/3558870661 shutdown_connections 2024-04-25T14:03:33.552 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.548+0000 7f557d414640 1 -- 172.21.15.150:0/3558870661 wait complete. 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.550+0000 7f557d414640 1 Processor -- start 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.550+0000 7f557d414640 1 -- start start 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f557806d630 0x7f55780cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55780b91f0 0x7f5578161c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f557806c8f0 con 0x7f55780b4c10 2024-04-25T14:03:33.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f557806c610 con 0x7f55780b91f0 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f557d414640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f557806da40 con 0x7f557806d630 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45248/0 (socket says 172.21.15.150:45248) 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 -- 172.21.15.150:0/2552370022 learned_addr learned my addr 172.21.15.150:0/2552370022 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f5576ffd640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f557806d630 0x7f55780cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55777fe640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55780b91f0 0x7f5578161c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 -- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f557806d630 msgr2=0x7f55780cfe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f557806d630 0x7f55780cfe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 -- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55780b91f0 msgr2=0x7f5578161c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55780b91f0 0x7f5578161c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 -- 172.21.15.150:0/2552370022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f557814db30 con 0x7f55780b4c10 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.552+0000 7f55767fc640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f556c02a290 tx=0x7f556c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.553+0000 7f555f7fe640 1 -- 172.21.15.150:0/2552370022 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c041020 con 0x7f55780b4c10 2024-04-25T14:03:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.553+0000 7f555f7fe640 1 -- 172.21.15.150:0/2552370022 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c0306c0 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.553+0000 7f555f7fe640 1 -- 172.21.15.150:0/2552370022 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c037040 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.553+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f557814de10 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.553+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55780acc70 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.554+0000 7f555f7fe640 1 -- 172.21.15.150:0/2552370022 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f556c044020 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.555+0000 7f555f7fe640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 0x7f55640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.555+0000 7f555f7fe640 1 -- 172.21.15.150:0/2552370022 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f556c0b59b0 con 0x7f55780b4c10 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.555+0000 7f5576ffd640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 0x7f55640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.555+0000 7f5576ffd640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 0x7f55640769c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f55680079a0 tx=0x7f5568008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.556+0000 7f558439b740 1 Processor -- start 2024-04-25T14:03:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.556+0000 7f558439b740 1 -- start start 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daa98600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6a33b0 0x55c7daa94fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c7da9d2740 con 0x55c7da6ab590 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55c7da96e3a0 con 0x55c7da6a33b0 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55c7da9ce700 con 0x55c7da6cc270 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f55777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f55777fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45252/0 (socket says 172.21.15.150:45252) 2024-04-25T14:03:33.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.557+0000 7f55777fe640 1 -- 172.21.15.150:0/3330054745 learned_addr learned my addr 172.21.15.150:0/3330054745 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55767fc640 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6a33b0 0x55c7daa94fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f5576ffd640 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daa98600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 -- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 msgr2=0x55c7daa98600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daa98600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 -- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6a33b0 msgr2=0x55c7daa94fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6a33b0 0x55c7daa94fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 -- 172.21.15.150:0/3330054745 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7da98a750 con 0x55c7da6ab590 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f55777fe640 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5570001ec0 tx=0x7f55700015d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c061bec8a0e51548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f555cff9640 1 -- 172.21.15.150:0/3330054745 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557000fdf0 con 0x55c7da6ab590 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f555cff9640 1 -- 172.21.15.150:0/3330054745 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5570018070 con 0x55c7da6ab590 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.558+0000 7f555cff9640 1 -- 172.21.15.150:0/3330054745 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5570003b70 con 0x55c7da6ab590 2024-04-25T14:03:33.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 -- 172.21.15.150:0/3330054745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 msgr2=0x55c7daa6e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5570001ec0 tx=0x7f55700015d0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 -- 172.21.15.150:0/3330054745 shutdown_connections 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa6e260 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6a33b0 0x55c7daa94fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 --2- 172.21.15.150:0/3330054745 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daa98600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 -- 172.21.15.150:0/3330054745 >> 172.21.15.150:0/3330054745 conn(0x55c7da6c03c0 msgr2=0x55c7daa6d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 -- 172.21.15.150:0/3330054745 shutdown_connections 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.559+0000 7f558439b740 1 -- 172.21.15.150:0/3330054745 wait complete. 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.560+0000 7f558439b740 1 -- start start 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.560+0000 7f558439b740 1 Processor -- start 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daac2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa94840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c7da959fe0 con 0x55c7da6ab590 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55c7daa9b940 con 0x55c7da6cc270 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55c7daab1bd0 con 0x55c7da6a33b0 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f558439b740 10 client.0 _ll_register_callbacks cb 0x55c7da970150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45254/0 (socket says 172.21.15.150:45254) 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.561+0000 7f55767fc640 1 -- 172.21.15.150:0/405325484 learned_addr learned my addr 172.21.15.150:0/405325484 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5576ffd640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daac2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55777fe640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa94840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 -- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 msgr2=0x55c7daac2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daac2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 -- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 msgr2=0x55c7daa94840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa94840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 -- 172.21.15.150:0/405325484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7da6e42d0 con 0x55c7da6ab590 2024-04-25T14:03:33.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5576ffd640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daac2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f55767fc640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f556c07c070 tx=0x7f556c07f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feaa1eec2b03cd39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0b5040 con 0x55c7da6ab590 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f556c088310 con 0x55c7da6ab590 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5570018070 con 0x55c7da6ab590 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f556c0b3050 con 0x55c7da6ab590 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 10 client.5562 fetch_fsmap learned FSMap version 16 2024-04-25T14:03:33.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55c7da7ae6b0 con 0x55c7da6ab590 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5570018070 con 0x55c7da6ab590 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f556c03a030 con 0x55c7da6ab590 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 10 client.5562 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 10 client.5562 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.562+0000 7f558439b740 20 client.5562 populate_metadata read hostname 'smithi150' 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7da68ae40 0x55c7daa995c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f558439b740 4 client.5562 mds_command: new command op to 5349 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 --> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7da9e0350 con 0x55c7da68ae40 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f5576ffd640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7da68ae40 0x55c7daa995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f5555ffb640 10 client.5562 ms_handle_connect on v2:172.21.15.150:6834/3196344346 2024-04-25T14:03:33.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.563+0000 7f5576ffd640 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7da68ae40 0x55c7daa995c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb0f6146713bfef2 server_cookie=efdc11db72c4441f in_seq=0 out_seq=0 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f5555ffb640 1 -- 172.21.15.150:0/405325484 <== mds.0 v2:172.21.15.150:6834/3196344346 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c7da9e0350 con 0x55c7da68ae40 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f5555ffb640 10 client.5562 handle_command_reply: tid=0 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 client.5562 shutdown 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 msgr2=0x55c7daa94300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f556c07c070 tx=0x7f556c07f1f0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 shutdown_connections 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7da68ae40 0x55c7daa995c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa94840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa94300 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 --2- 172.21.15.150:0/405325484 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daac2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f5555ffb640 0 client.5562 ms_handle_reset on v2:172.21.15.150:6834/3196344346 2024-04-25T14:03:33.572 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 >> 172.21.15.150:0/405325484 conn(0x55c7da6c03c0 msgr2=0x55c7daa92160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 shutdown_connections 2024-04-25T14:03:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 1 -- 172.21.15.150:0/405325484 wait complete. 2024-04-25T14:03:33.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.564+0000 7f558439b740 20 client.5562 trim_cache size 0 max 16384 2024-04-25T14:03:33.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.583+0000 7f558439b740 1 Processor -- start 2024-04-25T14:03:33.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.583+0000 7f558439b740 1 -- start start 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daa98600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa8ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5578203400 con 0x55c7da6ab590 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5578202400 con 0x55c7da6cc270 2024-04-25T14:03:33.586 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f55781fa620 con 0x55c7da6a33b0 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f55767fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45270/0 (socket says 172.21.15.150:45270) 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.584+0000 7f55767fc640 1 -- 172.21.15.150:0/2381148343 learned_addr learned my addr 172.21.15.150:0/2381148343 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55777fe640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa8ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 -- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 msgr2=0x55c7daa98600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f5576ffd640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daa98600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daa98600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 -- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 msgr2=0x55c7daa8ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa8ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 -- 172.21.15.150:0/2381148343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7daa910e0 con 0x55c7da6ab590 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55777fe640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa8ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f5576ffd640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daa98600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f55767fc640 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f556c07bb00 tx=0x7f556c008000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcb11b0eac0e54ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f5557fff640 1 -- 172.21.15.150:0/2381148343 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0887a0 con 0x55c7da6ab590 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f5557fff640 1 -- 172.21.15.150:0/2381148343 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c088310 con 0x55c7da6ab590 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.585+0000 7f5557fff640 1 -- 172.21.15.150:0/2381148343 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c07ece0 con 0x55c7da6ab590 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- 172.21.15.150:0/2381148343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 msgr2=0x55c7daa98b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f556c07bb00 tx=0x7f556c008000 comp rx=0 tx=0).stop 2024-04-25T14:03:33.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- 172.21.15.150:0/2381148343 shutdown_connections 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6cc270 0x55c7daa8ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6ab590 0x55c7daa98b40 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 --2- 172.21.15.150:0/2381148343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6a33b0 0x55c7daa98600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- 172.21.15.150:0/2381148343 >> 172.21.15.150:0/2381148343 conn(0x55c7da6c03c0 msgr2=0x55c7daa68bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- 172.21.15.150:0/2381148343 shutdown_connections 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- 172.21.15.150:0/2381148343 wait complete. 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.586+0000 7f558439b740 1 -- start start 2024-04-25T14:03:33.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.587+0000 7f558439b740 1 Processor -- start 2024-04-25T14:03:33.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6ab590 0x55c7daacb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daabe3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c7daab21a0 con 0x55c7da6a33b0 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55c7daa9b680 con 0x55c7da6ab590 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55c7daa9ad00 con 0x55c7da6cc270 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 10 client.0 _ll_register_callbacks cb 0x7f556c0094d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45282/0 (socket says 172.21.15.150:45282) 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 -- 172.21.15.150:0/704825577 learned_addr learned my addr 172.21.15.150:0/704825577 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f55767fc640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6ab590 0x55c7daacb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f55777fe640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daabe3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 -- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 msgr2=0x55c7daabe3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daabe3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 -- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6ab590 msgr2=0x55c7daacb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6ab590 0x55c7daacb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 -- 172.21.15.150:0/704825577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7daa9a1c0 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f55777fe640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daabe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5576ffd640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f55680073e0 tx=0x7f5568004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1391d4eec81e2fd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5555ffb640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556801c040 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f556800f040 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.588+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f556c088310 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f5568037050 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 10 client.5568 fetch_fsmap learned FSMap version 16 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55c7daa99520 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f556c088310 con 0x55c7da6a33b0 2024-04-25T14:03:33.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 16) ==== 3494+0+0 (secure 0 0 0) 0x7f556803a020 con 0x55c7da6a33b0 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 10 client.5568 fetch_fsmap finished waiting for FSMap version 16 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 10 client.5568 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 20 client.5568 populate_metadata read hostname 'smithi150' 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7daac2040 0x55c7daac2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 4 client.5568 mds_command: new command op to 5349 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 --> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] -- command(tid 0: {"prefix": "status"}) -- 0x55c7da9da1c0 con 0x55c7daac2040 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f55767fc640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7daac2040 0x55c7daac2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f5555ffb640 10 client.5568 ms_handle_connect on v2:172.21.15.150:6834/3196344346 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.589+0000 7f55767fc640 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7daac2040 0x55c7daac2450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ae8a91c95cc7fb4 server_cookie=aa291aac388a758 in_seq=0 out_seq=0 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f5555ffb640 1 -- 172.21.15.150:0/704825577 <== mds.0 v2:172.21.15.150:6834/3196344346 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55c7da9da1c0 con 0x55c7daac2040 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f5555ffb640 10 client.5568 handle_command_reply: tid=0 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 client.5568 shutdown 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 msgr2=0x55c7dab301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f55680073e0 tx=0x7f5568004730 comp rx=0 tx=0).stop 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 shutdown_connections 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:6834/3196344346,v1:172.21.15.150:6836/3196344346] conn(0x55c7daac2040 0x55c7daac2450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c7da6cc270 0x55c7daabe3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f5555ffb640 0 client.5568 ms_handle_reset on v2:172.21.15.150:6834/3196344346 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c7da6ab590 0x55c7daacb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 --2- 172.21.15.150:0/704825577 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c7da6a33b0 0x55c7dab301e0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 >> 172.21.15.150:0/704825577 conn(0x55c7da6c03c0 msgr2=0x55c7daab5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 shutdown_connections 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 1 -- 172.21.15.150:0/704825577 wait complete. 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f558439b740 20 client.5568 trim_cache size 0 max 16384 2024-04-25T14:03:33.597 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5349, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 10.122814717000001, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 16, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 38, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 38, 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 17.008747398000001 2024-04-25T14:03:33.598 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:03:33.599 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 msgr2=0x7f55640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 0x7f55640769c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f55680079a0 tx=0x7f5568008040 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 msgr2=0x7f5578161740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f556c02a290 tx=0x7f556c0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 shutdown_connections 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55640744b0 0x7f55640769c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f55780b91f0 0x7f5578161c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f55780b4c10 0x7f5578161740 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 --2- 172.21.15.150:0/2552370022 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f557806d630 0x7f55780cfe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.590+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 >> 172.21.15.150:0/2552370022 conn(0x7f55780b7690 msgr2=0x7f55780b59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.591+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 shutdown_connections 2024-04-25T14:03:33.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.591+0000 7f557d414640 1 -- 172.21.15.150:0/2552370022 wait complete. 2024-04-25T14:03:33.602 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:03:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.710+0000 7fce52c4c640 1 Processor -- start 2024-04-25T14:03:33.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.710+0000 7fce52c4c640 1 -- start start 2024-04-25T14:03:33.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce52c4c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c156cd0 0x7fce4c15d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce52c4c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce4c070e70 0x7fce4c06ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce52c4c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce4c06d750 con 0x7fce4c06d340 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce52c4c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fce4c070250 con 0x7fce4c070e70 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce52c4c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c0afe00 con 0x7fce4c156cd0 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce4c070e70 0x7fce4c06ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fce4c070e70 0x7fce4c06ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45296/0 (socket says 172.21.15.150:45296) 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 -- 172.21.15.150:0/1188321731 learned_addr learned my addr 172.21.15.150:0/1188321731 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce509e8640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c156cd0 0x7fce4c15d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 -- 172.21.15.150:0/1188321731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c156cd0 msgr2=0x7fce4c15d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c156cd0 0x7fce4c15d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 -- 172.21.15.150:0/1188321731 >> v1:172.21.15.187:6789/0 conn(0x7fce4c06d340 legacy=0x7fce4c160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.712+0000 7fce4bfff640 1 -- 172.21.15.150:0/1188321731 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c0b00a0 con 0x7fce4c070e70 2024-04-25T14:03:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.713+0000 7fce4bfff640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c070e70 0x7fce4c06ce00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fce4000aa10 tx=0x7fce4002b610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b0a1c909d95665 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.713+0000 7fce4b7fe640 1 -- 172.21.15.150:0/1188321731 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4000b040 con 0x7fce4c070e70 2024-04-25T14:03:33.713 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.713+0000 7fce4b7fe640 1 -- 172.21.15.150:0/1188321731 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce40030710 con 0x7fce4c070e70 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.713+0000 7fce4b7fe640 1 -- 172.21.15.150:0/1188321731 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40030ae0 con 0x7fce4c070e70 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 -- 172.21.15.150:0/1188321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c070e70 msgr2=0x7fce4c06ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c070e70 0x7fce4c06ce00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fce4000aa10 tx=0x7fce4002b610 comp rx=0 tx=0).stop 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 -- 172.21.15.150:0/1188321731 shutdown_connections 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c070e70 0x7fce4c06ce00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1188321731 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c156cd0 0x7fce4c15d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 -- 172.21.15.150:0/1188321731 >> 172.21.15.150:0/1188321731 conn(0x7fce4c0d1ae0 msgr2=0x7fce4c0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 -- 172.21.15.150:0/1188321731 shutdown_connections 2024-04-25T14:03:33.714 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.714+0000 7fce52c4c640 1 -- 172.21.15.150:0/1188321731 wait complete. 2024-04-25T14:03:33.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.716+0000 7fce52c4c640 1 Processor -- start 2024-04-25T14:03:33.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.716+0000 7fce52c4c640 1 -- start start 2024-04-25T14:03:33.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c070e70 0x7fce4c149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 0x7fce4c14a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c0b0940 con 0x7fce4c06d340 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fce4c0b0660 con 0x7fce4c156cd0 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce52c4c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fce4c0b0e60 con 0x7fce4c070e70 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45302/0 (socket says 172.21.15.150:45302) 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 -- 172.21.15.150:0/1338240318 learned_addr learned my addr 172.21.15.150:0/1338240318 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce511e9640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 0x7fce4c14a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce4bfff640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c070e70 0x7fce4c149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 -- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c070e70 msgr2=0x7fce4c149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c070e70 0x7fce4c149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 -- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 msgr2=0x7fce4c14a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.718 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 0x7fce4c14a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce509e8640 1 -- 172.21.15.150:0/1338240318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4c14c1c0 con 0x7fce4c06d340 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.717+0000 7fce511e9640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 0x7fce4c14a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce509e8640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d600 tx=0x7fce3c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c004240 con 0x7fce4c06d340 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce3c0043e0 con 0x7fce4c06d340 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4c141080 con 0x7fce4c06d340 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c003830 con 0x7fce4c06d340 2024-04-25T14:03:33.719 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.718+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce4c141320 con 0x7fce4c06d340 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.719+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce4c076a40 con 0x7fce4c06d340 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.719+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fce3c0039d0 con 0x7fce4c06d340 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.720+0000 7fce497fa640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 0x7fce380769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.720+0000 7fce4bfff640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 0x7fce380769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.720+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fce3c095580 con 0x7fce4c06d340 2024-04-25T14:03:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.720+0000 7fce4bfff640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 0x7fce380769c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fce40035000 tx=0x7fce40030000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:33.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.725+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fce3c05fc10 con 0x7fce4c06d340 2024-04-25T14:03:33.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.937+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fce4c0b61d0 con 0x7fce4c06d340 2024-04-25T14:03:33.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.938+0000 7fce497fa640 1 -- 172.21.15.150:0/1338240318 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7fce3c05f510 con 0x7fce4c06d340 2024-04-25T14:03:33.937 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 msgr2=0x7fce380769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 0x7fce380769c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fce40035000 tx=0x7fce40030000 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 msgr2=0x7fce4c14bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00d600 tx=0x7fce3c00dac0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 shutdown_connections 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fce380744b0 0x7fce380769c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fce4c156cd0 0x7fce4c14a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fce4c070e70 0x7fce4c149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.939+0000 7fce52c4c640 1 --2- 172.21.15.150:0/1338240318 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fce4c06d340 0x7fce4c14bc80 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.940+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 >> 172.21.15.150:0/1338240318 conn(0x7fce4c0d1ae0 msgr2=0x7fce4c15f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:33.939 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.940+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 shutdown_connections 2024-04-25T14:03:33.940 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:33.940+0000 7fce52c4c640 1 -- 172.21.15.150:0/1338240318 wait complete. 2024-04-25T14:03:33.950 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T14:03:33.950 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:03:33.950 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-04-25T14:03:33.950 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T14:03:33.950 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T14:03:33.951 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T14:03:33.951 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T14:03:33.951 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:03:33.951 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:03:33.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:33.951 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-04-25T14:03:33.966 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:03:33.967 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42686sec preferred_lft 42686sec 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::eb5b:7bab:e07b:2aac/64 scope link noprefixroute 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 6a:dc:79:80:07:d5 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::e835:8ff:febc:ce5a/64 scope link 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fe:50:c0:ba:66:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T14:03:33.968 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::fc50:c0ff:feba:66ff/64 scope link 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 6a:dc:79:80:07:d5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::68dc:79ff:fe80:7d5/64 scope link 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:03:33.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:33.969 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:03:34.029 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-25T14:03:34.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:34.030 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-04-25T14:03:34.086 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-04-25T14:03:34.086 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:03:34.087 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:03:34.087 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:03:34.087 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:03:34.087 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:03:34.175 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:03:34.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:03:34.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:34.211 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:03:34.225 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T14:03:34.226 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T14:03:34.226 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:03:34.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:34.226 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-04-25T14:03:34.290 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:03:34.290 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T14:03:34.345 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:03:34.345 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:03:34.345 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:03:34.399 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T14:03:34.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:34.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:03:34.492 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:03:34.492 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:03:34.505 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:03:34.505 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:03:34.505 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:03:34.505 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with new device syntax: 0@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:03:34.505 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:03:34.506 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T14:03:34.506 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:03:34.506 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:03:34.507 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:03:34.507 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:03:34.508 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:03:34.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:34.509 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:03:34.532 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:03:34.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:03:34.627 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:03:34.633 INFO:teuthology.orchestra.run.smithi150.stdout:client5027 2024-04-25T14:03:34.634 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:34.634 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:34.651 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:03:34.693 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-25T14:03:34.731 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-25T14:03:34.732 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.733 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-25T14:03:34.734 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.735 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-25T14:03:34.736 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:03:34.737 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-25T14:03:34.738 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:03:34.739 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:03:34.740 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:03:34.741 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:03:34.742 INFO:teuthology.orchestra.run.smithi150.stdout: osd_deep_scrub_large_omap_object_key_threshold = 5 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-25T14:03:34.743 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.744 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:03:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:03:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-25T14:03:34.745 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.746 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-25T14:03:34.747 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-25T14:03:34.748 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-25T14:03:34.749 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-25T14:03:34.750 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:03:34.751 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:03:34.752 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:03:34.753 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:03:34.754 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-25T14:03:34.755 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:03:34.756 INFO:teuthology.orchestra.run.smithi187.stdout: osd_deep_scrub_large_omap_object_key_threshold = 5 2024-04-25T14:03:34.758 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T14:03:34.758 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T14:03:34.758 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T14:03:34.759 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:34.759 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T14:03:34.759 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T14:03:34.759 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T14:03:34.759 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:34.759 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T14:03:34.760 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T14:03:34.760 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T14:03:34.760 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:34.760 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T14:03:34.760 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T14:03:34.760 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T14:03:34.760 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:03:34.809 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T14:03:34.809 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T14:03:34.810 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T14:03:34.810 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T14:03:34.811 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T14:03:34.811 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T14:03:34.812 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T14:03:34.812 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T14:03:34.851 INFO:tasks.ceph.mds.c:Started 2024-04-25T14:03:34.852 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:34.852 INFO:tasks.ceph.mds.a:Started 2024-04-25T14:03:34.852 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:34.854 INFO:tasks.ceph.mds.b:Started 2024-04-25T14:03:34.855 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:34.855 INFO:tasks.ceph.mds.d:Started 2024-04-25T14:03:34.856 DEBUG:teuthology.parallel:result is None 2024-04-25T14:03:34.856 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:34.938 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-04-25T14:03:34.939 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:03:34.943 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:03:34.944+0000 7f46d47d6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:03:34.943 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:03:34.943 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:03:34.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.973+0000 7fa7a18e7640 1 Processor -- start 2024-04-25T14:03:34.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.973+0000 7fa7a18e7640 1 -- start start 2024-04-25T14:03:34.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa7a18e7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa79c0d4840 0x7fa79c0d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa7a18e7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa79c06c960 0x7fa79c06cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa7a18e7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa79c0d5190 con 0x7fa79c164710 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa7a18e7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa79c160940 con 0x7fa79c0d4840 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa7a18e7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa79c160c00 con 0x7fa79c06c960 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa793fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa79c0d4840 0x7fa79c0d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa79affd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa79c164710 0x7fa79c160230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:34518/0 (socket says 172.21.15.150:34518) 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa79affd640 1 -- 172.21.15.150:0/3234760543 learned_addr learned my addr 172.21.15.150:0/3234760543 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa79b7fe640 1 --2- 172.21.15.150:0/3234760543 >> v2:172.21.15.187:3300/0 conn(0x7fa79c06c960 0x7fa79c06cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa793fff640 1 -- 172.21.15.150:0/3234760543 >> v2:172.21.15.187:3300/0 conn(0x7fa79c06c960 msgr2=0x7fa79c06cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa793fff640 1 --2- 172.21.15.150:0/3234760543 >> v2:172.21.15.187:3300/0 conn(0x7fa79c06c960 0x7fa79c06cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa793fff640 1 -- 172.21.15.150:0/3234760543 >> v1:172.21.15.150:6789/0 conn(0x7fa79c164710 legacy=0x7fa79c160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:34.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.975+0000 7fa793fff640 1 -- 172.21.15.150:0/3234760543 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c160ee0 con 0x7fa79c0d4840 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.976+0000 7fa793fff640 1 --2- 172.21.15.150:0/3234760543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0d4c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0d5b50 tx=0x7fa78402a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0b4c2d8d4126a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.976+0000 7fa79a7fc640 1 -- 172.21.15.150:0/3234760543 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784037070 con 0x7fa79c0d4840 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.976+0000 7fa79a7fc640 1 -- 172.21.15.150:0/3234760543 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa78402ad30 con 0x7fa79c0d4840 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.976+0000 7fa79a7fc640 1 -- 172.21.15.150:0/3234760543 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784032660 con 0x7fa79c0d4840 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 -- 172.21.15.150:0/3234760543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 msgr2=0x7fa79c0d4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/3234760543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0d4c50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0d5b50 tx=0x7fa78402a0a0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 -- 172.21.15.150:0/3234760543 shutdown_connections 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/3234760543 >> v2:172.21.15.187:3300/0 conn(0x7fa79c06c960 0x7fa79c06cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/3234760543 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0d4c50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 -- 172.21.15.150:0/3234760543 >> 172.21.15.150:0/3234760543 conn(0x7fa79c0b77e0 msgr2=0x7fa79c0b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 -- 172.21.15.150:0/3234760543 shutdown_connections 2024-04-25T14:03:34.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.977+0000 7fa7a18e7640 1 -- 172.21.15.150:0/3234760543 wait complete. 2024-04-25T14:03:34.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.979+0000 7fa7a18e7640 1 Processor -- start 2024-04-25T14:03:34.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.979+0000 7fa7a18e7640 1 -- start start 2024-04-25T14:03:34.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 0x7fa79c0ba740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 0x7fa79c0c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c0d5630 con 0x7fa79c0d4840 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa79c1614a0 con 0x7fa79c06c960 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa7a18e7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa79c164b20 con 0x7fa79c164710 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45358/0 (socket says 172.21.15.150:45358) 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa793fff640 1 -- 172.21.15.150:0/1682904992 learned_addr learned my addr 172.21.15.150:0/1682904992 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.980+0000 7fa79b7fe640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 0x7fa79c0c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa79affd640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 0x7fa79c0ba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 -- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 msgr2=0x7fa79c0c9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:34.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 0x7fa79c0c9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 -- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 msgr2=0x7fa79c0ba740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 0x7fa79c0ba740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 -- 172.21.15.150:0/1682904992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa79c1f6fd0 con 0x7fa79c0d4840 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa79b7fe640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 0x7fa79c0c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa79affd640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 0x7fa79c0ba740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa793fff640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa78402a0e0 tx=0x7fa7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784041020 con 0x7fa79c0d4840 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa78403d420 con 0x7fa79c0d4840 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa79c1f72b0 con 0x7fa79c0d4840 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa784037040 con 0x7fa79c0d4840 2024-04-25T14:03:34.982 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.981+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa79c1f7870 con 0x7fa79c0d4840 2024-04-25T14:03:34.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.982+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa79c050740 con 0x7fa79c0d4840 2024-04-25T14:03:34.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.983+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa784044020 con 0x7fa79c0d4840 2024-04-25T14:03:34.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.983+0000 7fa798ff9640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 0x7fa77c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:34.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.983+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6187+0+0 (secure 0 0 0) 0x7fa7840b56f0 con 0x7fa79c0d4840 2024-04-25T14:03:34.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.983+0000 7fa79affd640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 0x7fa77c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:34.984 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.984+0000 7fa79affd640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 0x7fa77c0769f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa78c000f60 tx=0x7fa78c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:34.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:34.988+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa78404b030 con 0x7fa79c0d4840 2024-04-25T14:03:35.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.203+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa79c0d4110 con 0x7fa79c0d4840 2024-04-25T14:03:35.202 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.203+0000 7fa798ff9640 1 -- 172.21.15.150:0/1682904992 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+5181 (secure 0 0 0) 0x7fa784086e20 con 0x7fa79c0d4840 2024-04-25T14:03:35.203 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:35.203 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":18,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4889,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/21568960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":21568960},{"type":"v1","addr":"172.21.15.187:6834","nonce":21568960}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":18,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:34.949412+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":39,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5185},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5185":{"gid":5185,"name":"b","rank":0,"incarnation":18,"state":"up:replay","state_seq":1,"addr":"172.21.15.187:6835/1241446695","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":1241446695},{"type":"v1","addr":"172.21.15.187:6835","nonce":1241446695}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T14:03:35.203 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 18 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 msgr2=0x7fa77c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 0x7fa77c0769f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa78c000f60 tx=0x7fa78c009040 comp rx=0 tx=0).stop 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 msgr2=0x7fa79c0bac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa78402a0e0 tx=0x7fa7840046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 shutdown_connections 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa77c0744e0 0x7fa77c0769f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa79c164710 0x7fa79c0c9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:35.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa79c0d4840 0x7fa79c0bac80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:35.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 --2- 172.21.15.150:0/1682904992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa79c06c960 0x7fa79c0ba740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:35.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 >> 172.21.15.150:0/1682904992 conn(0x7fa79c0b77e0 msgr2=0x7fa79c15e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:35.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 shutdown_connections 2024-04-25T14:03:35.206 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:35.205+0000 7fa7a18e7640 1 -- 172.21.15.150:0/1682904992 wait complete. 2024-04-25T14:03:35.619 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 18, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:03:22.813096+0000', 'modified': '2024-04-25T14:03:34.949412+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 39, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5185}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5185': {'gid': 5185, 'name': 'b', 'rank': 0, 'incarnation': 18, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.187:6835/1241446695', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 1241446695}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 1241446695}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T14:03:35.619 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5185:up:replay 2024-04-25T14:03:35.676 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:03:35.677+0000 7f46d1fd1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:03:36.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.744+0000 7ff9ddca7640 1 Processor -- start 2024-04-25T14:03:36.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.744+0000 7ff9ddca7640 1 -- start start 2024-04-25T14:03:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80cebe0 0x7ff9d81648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9d80b45f0 0x7ff9d80b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff9d80d1ae0 0x7ff9d80d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d80d0300 con 0x7ff9d80b45f0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d8164e30 con 0x7ff9d80d1ae0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9ddca7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d80b4160 con 0x7ff9d80cebe0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9d80b45f0 0x7ff9d80b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9d80b45f0 0x7ff9d80b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45372/0 (socket says 172.21.15.150:45372) 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2158398457 learned_addr learned my addr 172.21.15.150:0/2158398457 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d7fff640 1 --2- 172.21.15.150:0/2158398457 >> v2:172.21.15.187:3300/0 conn(0x7ff9d80d1ae0 0x7ff9d80d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d77fe640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80cebe0 0x7ff9d81648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2158398457 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80cebe0 msgr2=0x7ff9d81648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80cebe0 0x7ff9d81648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2158398457 >> v2:172.21.15.187:3300/0 conn(0x7ff9d80d1ae0 msgr2=0x7ff9d80d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2158398457 >> v2:172.21.15.187:3300/0 conn(0x7ff9d80d1ae0 0x7ff9d80d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.746+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2158398457 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d8160810 con 0x7ff9d80b45f0 2024-04-25T14:03:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.747+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80b45f0 0x7ff9d80b4a00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009870 tx=0x7ff9cc02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3f9a04d233272a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.747+0000 7ff9d67fc640 1 -- 172.21.15.150:0/2158398457 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc037070 con 0x7ff9d80b45f0 2024-04-25T14:03:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.747+0000 7ff9d67fc640 1 -- 172.21.15.150:0/2158398457 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9cc032440 con 0x7ff9d80b45f0 2024-04-25T14:03:36.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.747+0000 7ff9d67fc640 1 -- 172.21.15.150:0/2158398457 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc032810 con 0x7ff9d80b45f0 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2158398457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80b45f0 msgr2=0x7ff9d80b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80b45f0 0x7ff9d80b4a00 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc009870 tx=0x7ff9cc02a210 comp rx=0 tx=0).stop 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2158398457 shutdown_connections 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2158398457 >> v2:172.21.15.187:3300/0 conn(0x7ff9d80d1ae0 0x7ff9d80d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80b45f0 0x7ff9d80b4a00 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2158398457 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80cebe0 0x7ff9d81648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2158398457 >> 172.21.15.150:0/2158398457 conn(0x7ff9d80b76d0 msgr2=0x7ff9d80b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:36.749 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2158398457 shutdown_connections 2024-04-25T14:03:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.748+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2158398457 wait complete. 2024-04-25T14:03:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.750+0000 7ff9ddca7640 1 Processor -- start 2024-04-25T14:03:36.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.750+0000 7ff9ddca7640 1 -- start start 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9d80b45f0 0x7ff9d8155c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80d1ae0 0x7ff9d8162740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80cfb30 con 0x7ff9d80cebe0 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d80cfe10 con 0x7ff9d80b45f0 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9ddca7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d80b3df0 con 0x7ff9d80d1ae0 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9d6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:45384/0 (socket says 172.21.15.150:45384) 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2091596032 learned_addr learned my addr 172.21.15.150:0/2091596032 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:36.752 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9d77fe640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9d80b45f0 0x7ff9d8155c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.751+0000 7ff9d7fff640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80d1ae0 0x7ff9d8162740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80d1ae0 msgr2=0x7ff9d8162740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80d1ae0 0x7ff9d8162740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9d80b45f0 msgr2=0x7ff9d8155c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9d80b45f0 0x7ff9d8155c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 -- 172.21.15.150:0/2091596032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d80b4f40 con 0x7ff9d80cebe0 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9d6ffd640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc000c00 tx=0x7ff9cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc041020 con 0x7ff9d80cebe0 2024-04-25T14:03:36.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9cc03d620 con 0x7ff9d80cebe0 2024-04-25T14:03:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d81616a0 con 0x7ff9d80cebe0 2024-04-25T14:03:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc037040 con 0x7ff9d80cebe0 2024-04-25T14:03:36.754 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.752+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d8161c60 con 0x7ff9d80cebe0 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.753+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d8050740 con 0x7ff9d80cebe0 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.754+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff9cc044020 con 0x7ff9d80cebe0 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.754+0000 7ff9bffff640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 0x7ff9c40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.754+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7ff9cc0b5990 con 0x7ff9d80cebe0 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.754+0000 7ff9d77fe640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 0x7ff9c40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:36.755 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.755+0000 7ff9d77fe640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 0x7ff9c40769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80b1fc0 tx=0x7ff9c8009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:36.758 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.759+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9cc04b030 con 0x7ff9d80cebe0 2024-04-25T14:03:36.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.970+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff9d8072310 con 0x7ff9d80cebe0 2024-04-25T14:03:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.971+0000 7ff9bffff640 1 -- 172.21.15.150:0/2091596032 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) ==== 94+0+5190 (secure 0 0 0) 0x7ff9cc07f920 con 0x7ff9d80cebe0 2024-04-25T14:03:36.970 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:36.970 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":21,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":21,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:35.697988+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:reconnect","state_seq":2,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T14:03:36.970 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 21 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 msgr2=0x7ff9c40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 0x7ff9c40769c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff9d80b1fc0 tx=0x7ff9c8009110 comp rx=0 tx=0).stop 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 msgr2=0x7ff9d809eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc000c00 tx=0x7ff9cc0046c0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 shutdown_connections 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9c40744b0 0x7ff9c40769c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff9d80d1ae0 0x7ff9d8162740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9d80cebe0 0x7ff9d809eb10 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 --2- 172.21.15.150:0/2091596032 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff9d80b45f0 0x7ff9d8155c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 >> 172.21.15.150:0/2091596032 conn(0x7ff9d80b76d0 msgr2=0x7ff9d815fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 shutdown_connections 2024-04-25T14:03:36.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:36.973+0000 7ff9ddca7640 1 -- 172.21.15.150:0/2091596032 wait complete. 2024-04-25T14:03:37.386 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 21, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:03:22.813096+0000', 'modified': '2024-04-25T14:03:35.697988+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5374': {'gid': 5374, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:reconnect', 'state_seq': 2, 'addr': '172.21.15.187:6835/4048124689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 4048124689}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 4048124689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T14:03:37.386 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5374:up:reconnect 2024-04-25T14:03:38.386 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:38.446 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:03:38.449 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:03:38.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.512+0000 7fa5c8eb6640 1 Processor -- start 2024-04-25T14:03:38.512 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.513+0000 7fa5c8eb6640 1 -- start start 2024-04-25T14:03:38.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.514+0000 7fa5c8eb6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.514+0000 7fa5c8eb6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 0x7fa5c40d05e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.514+0000 7fa5c8eb6640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5c4160ce0 con 0x7fa5c41614a0 2024-04-25T14:03:38.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.514+0000 7fa5c8eb6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa5c40d0440 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.514+0000 7fa5c8eb6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c40b52d0 con 0x7fa5c40d30b0 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c259c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c2d9d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 0x7fa5c40d05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c259c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:34156/0 (socket says 172.21.15.150:34156) 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c259c640 1 -- 172.21.15.150:0/1986965794 learned_addr learned my addr 172.21.15.150:0/1986965794 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c2d9d640 1 -- 172.21.15.150:0/1986965794 >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 msgr2=0x7fa5c40b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c2d9d640 1 --2- 172.21.15.150:0/1986965794 >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c2d9d640 1 -- 172.21.15.150:0/1986965794 >> v1:172.21.15.187:6789/0 conn(0x7fa5c41614a0 legacy=0x7fa5c40d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c159a640 1 -- 172.21.15.150:0/1986965794 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1193611860 0 0) 0x7fa5c4160ce0 con 0x7fa5c41614a0 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c2d9d640 1 -- 172.21.15.150:0/1986965794 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c40cfea0 con 0x7fa5c40d30b0 2024-04-25T14:03:38.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.515+0000 7fa5c259c640 1 --2- 172.21.15.150:0/1986965794 >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:38.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.516+0000 7fa5c2d9d640 1 --2- 172.21.15.150:0/1986965794 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 0x7fa5c40d05e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0097a0 tx=0x7fa5bc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2db15bccf77cae1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:38.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.516+0000 7fa5c159a640 1 -- 172.21.15.150:0/1986965794 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc037070 con 0x7fa5c40d30b0 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.516+0000 7fa5c159a640 1 -- 172.21.15.150:0/1986965794 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5bc032440 con 0x7fa5c40d30b0 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.516+0000 7fa5c159a640 1 -- 172.21.15.150:0/1986965794 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5bc032810 con 0x7fa5c40d30b0 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1986965794 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 msgr2=0x7fa5c40d05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1986965794 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 0x7fa5c40d05e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc0097a0 tx=0x7fa5bc02a180 comp rx=0 tx=0).stop 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1986965794 shutdown_connections 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1986965794 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c40d30b0 0x7fa5c40d05e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1986965794 >> v2:172.21.15.187:3300/0 conn(0x7fa5c40b4ec0 0x7fa5c40b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1986965794 >> 172.21.15.150:0/1986965794 conn(0x7fa5c40b7720 msgr2=0x7fa5c40b7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1986965794 shutdown_connections 2024-04-25T14:03:38.517 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.517+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1986965794 wait complete. 2024-04-25T14:03:38.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.519+0000 7fa5c8eb6640 1 Processor -- start 2024-04-25T14:03:38.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.519+0000 7fa5c8eb6640 1 -- start start 2024-04-25T14:03:38.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.520 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 0x7fa5c40cbb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 0x7fa5c40cc090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c40cf900 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c4160f80 con 0x7fa5c40b1bb0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c8eb6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5c41618b0 con 0x7fa5c41614a0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49742/0 (socket says 172.21.15.150:49742) 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 -- 172.21.15.150:0/1501228623 learned_addr learned my addr 172.21.15.150:0/1501228623 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c1d9b640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 0x7fa5c40cbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c2d9d640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 0x7fa5c40cc090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 -- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 msgr2=0x7fa5c40cbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 0x7fa5c40cbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 -- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 msgr2=0x7fa5c40cc090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.521 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 0x7fa5c40cc090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c259c640 1 -- 172.21.15.150:0/1501228623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c40cc910 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c2d9d640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 0x7fa5c40cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.520+0000 7fa5c1d9b640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 0x7fa5c40cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5c259c640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e0f0 tx=0x7fa5b400e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b400b310 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5b4009600 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c40b2290 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40099b0 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.522 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.521+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c40c9ee0 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.522+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5c40c9cd0 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.523+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa5b4014050 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.523+0000 7fa5b2ffd640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 0x7fa5a40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.523+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7fa5b4097f10 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.523+0000 7fa5c1d9b640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 0x7fa5a40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:38.523 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.524+0000 7fa5c1d9b640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 0x7fa5a40769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5c406f370 tx=0x7fa5b8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:38.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.528+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5c40cc910 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.743+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5c4050740 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.743+0000 7fa5b2ffd640 1 -- 172.21.15.150:0/1501228623 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7fa5b4012040 con 0x7fa5c40b4ec0 2024-04-25T14:03:38.743 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:38.743 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:37.694103+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:clientreplay","state_seq":4,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-25T14:03:38.743 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 23 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 msgr2=0x7fa5a40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 0x7fa5a40769c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa5c406f370 tx=0x7fa5b8009040 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 msgr2=0x7fa5c4156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fa5b400e0f0 tx=0x7fa5b400e5b0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 shutdown_connections 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5a40744b0 0x7fa5a40769c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5c40b1bb0 0x7fa5c40cc090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5c41614a0 0x7fa5c40cbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 --2- 172.21.15.150:0/1501228623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5c40b4ec0 0x7fa5c4156bc0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 >> 172.21.15.150:0/1501228623 conn(0x7fa5c40b7720 msgr2=0x7fa5c415a8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 shutdown_connections 2024-04-25T14:03:38.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:38.745+0000 7fa5c8eb6640 1 -- 172.21.15.150:0/1501228623 wait complete. 2024-04-25T14:03:39.158 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:03:22.813096+0000', 'modified': '2024-04-25T14:03:37.694103+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5374': {'gid': 5374, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:clientreplay', 'state_seq': 4, 'addr': '172.21.15.187:6835/4048124689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 4048124689}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 4048124689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-25T14:03:39.159 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5374:up:clientreplay 2024-04-25T14:03:40.159 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:03:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.281+0000 7f9787677640 1 Processor -- start 2024-04-25T14:03:40.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.282+0000 7f9787677640 1 -- start start 2024-04-25T14:03:40.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9787677640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f978015e050 0x7f97800b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9787677640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97800b5e70 con 0x7f97800b5a60 2024-04-25T14:03:40.287 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9787677640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97800cebe0 con 0x7f9780161bd0 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9787677640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f97800d2080 con 0x7f978015e050 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9785c14640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f97800b5a60 0x7f97800d27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:40802/0 (socket says 172.21.15.150:40802) 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9785c14640 1 -- 172.21.15.150:0/221264136 learned_addr learned my addr 172.21.15.150:0/221264136 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9784c12640 1 --2- 172.21.15.150:0/221264136 >> v2:172.21.15.187:3300/0 conn(0x7f978015e050 0x7f97800b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4001922854 0 0) 0x7f9778001110 con 0x7f97800b5a60 2024-04-25T14:03:40.288 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.283+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f976c003520 con 0x7f97800b5a60 2024-04-25T14:03:40.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 979922299 0 0) 0x7f976c003520 con 0x7f97800b5a60 2024-04-25T14:03:40.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 >> v2:172.21.15.187:3300/0 conn(0x7f978015e050 msgr2=0x7f97800b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:40.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 --2- 172.21.15.150:0/221264136 >> v2:172.21.15.187:3300/0 conn(0x7f978015e050 0x7f97800b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 >> v1:172.21.15.187:6789/0 conn(0x7f9780161bd0 legacy=0x7f97800ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.289 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97800ceec0 con 0x7f97800b5a60 2024-04-25T14:03:40.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f9778001840 con 0x7f97800b5a60 2024-04-25T14:03:40.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 >> v1:172.21.15.150:6789/0 conn(0x7f97800b5a60 legacy=0x7f97800d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.290 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f976c0039b0 0x7f976c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c004800 0x7f976c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 0x7f976c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f97800d2080 con 0x7f976c005250 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f976c00c1e0 con 0x7f976c004800 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f976c00c3e0 con 0x7f976c0039b0 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 0x7f976c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9784c12640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c004800 0x7f976c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785413640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f976c0039b0 0x7f976c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 -- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f976c0039b0 msgr2=0x7f976c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f976c0039b0 0x7f976c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 -- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c004800 msgr2=0x7f976c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c004800 0x7f976c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.284+0000 7f9785c14640 1 -- 172.21.15.150:0/221264136 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f976c003520 con 0x7f976c005250 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.285+0000 7f9785c14640 1 -- 172.21.15.150:0/221264136 reap_dead start 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.285+0000 7f9785c14640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 0x7f976c00b9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f977800c4c0 tx=0x7f976c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4a53bd2a574289 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.285+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977800c0e0 con 0x7f976c005250 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.285+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97780058d0 con 0x7f976c005250 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.285+0000 7f9777fff640 1 -- 172.21.15.150:0/221264136 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778005ce0 con 0x7f976c005250 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 -- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 msgr2=0x7f976c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 0x7f976c00b9c0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f977800c4c0 tx=0x7f976c005830 comp rx=0 tx=0).stop 2024-04-25T14:03:40.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 -- 172.21.15.150:0/221264136 shutdown_connections 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 --2- 172.21.15.150:0/221264136 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f976c005250 0x7f976c00b9c0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 -- 172.21.15.150:0/221264136 >> 172.21.15.150:0/221264136 conn(0x7f97800d0520 msgr2=0x7f97800d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 -- 172.21.15.150:0/221264136 shutdown_connections 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.286+0000 7f9787677640 1 -- 172.21.15.150:0/221264136 wait complete. 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.288+0000 7f9787677640 1 Processor -- start 2024-04-25T14:03:40.293 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.288+0000 7f9787677640 1 -- start start 2024-04-25T14:03:40.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c005250 0x7f9780156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f978006f740 0x7f978006fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f978015e460 con 0x7f97801607b0 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f97800d22d0 con 0x7f976c005250 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9787677640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f97800b6d80 con 0x7f978006f740 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9785c14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9785c14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49766/0 (socket says 172.21.15.150:49766) 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9785c14640 1 -- 172.21.15.150:0/464785111 learned_addr learned my addr 172.21.15.150:0/464785111 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9785413640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c005250 0x7f9780156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.289+0000 7f9784c12640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f978006f740 0x7f978006fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 -- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f978006f740 msgr2=0x7f978006fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f978006f740 0x7f978006fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 -- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c005250 msgr2=0x7f9780156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c005250 0x7f9780156cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.296 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 -- 172.21.15.150:0/464785111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9780161100 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9785c14640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f977802afe0 tx=0x7f97800cedb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977803e030 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9778000ed0 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9778001110 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f978006e200 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.290+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f978006e4e0 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.291+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9780050740 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.292+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9778036040 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.292+0000 7f9775ffb640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 0x7f97640769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.292+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7f97780b5560 con 0x7f97801607b0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.292+0000 7f9785413640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 0x7f97640769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:40.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.293+0000 7f9785413640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 0x7f97640769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f977c000f80 tx=0x7f977c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:40.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.297+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9780161100 con 0x7f97801607b0 2024-04-25T14:03:40.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.510+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9780069970 con 0x7f97801607b0 2024-04-25T14:03:40.510 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.511+0000 7f9775ffb640 1 -- 172.21.15.150:0/464785111 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5194 (secure 0 0 0) 0x7f97800b5e70 con 0x7f97801607b0 2024-04-25T14:03:40.510 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:03:40.511 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:39.701192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-04-25T14:03:40.511 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 msgr2=0x7f97640769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 0x7f97640769c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f977c000f80 tx=0x7f977c009040 comp rx=0 tx=0).stop 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 msgr2=0x7f9780160bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f977802afe0 tx=0x7f97800cedb0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 shutdown_connections 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f97640744b0 0x7f97640769c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f97801607b0 0x7f9780160bc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f978006f740 0x7f978006fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 --2- 172.21.15.150:0/464785111 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f976c005250 0x7f9780156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:40.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 >> 172.21.15.150:0/464785111 conn(0x7f97800d0520 msgr2=0x7f97800b7a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:40.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 shutdown_connections 2024-04-25T14:03:40.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:40.513+0000 7f9787677640 1 -- 172.21.15.150:0/464785111 wait complete. 2024-04-25T14:03:40.688 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-25T14:03:40.688+0000 7f693e348640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:03:40.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:03:22.813096+0000', 'modified': '2024-04-25T14:03:39.701192+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5374}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5374': {'gid': 5374, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.187:6835/4048124689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 4048124689}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 4048124689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5374, 'qdb_cluster': [5374]} 2024-04-25T14:03:40.927 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:03:40.928 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T14:03:41.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.037+0000 7fee804e0640 1 Processor -- start 2024-04-25T14:03:41.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.037+0000 7fee804e0640 1 -- start start 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee804e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee804e0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 0x7fee780b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee804e0640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee780b5e70 con 0x7fee780b5a60 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee804e0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fee780d2080 con 0x7fee780ce390 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee804e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee780cebe0 con 0x7fee78161bd0 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7da7b640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 0x7fee780b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:34568/0 (socket says 172.21.15.150:34568) 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 -- 172.21.15.150:0/1032490144 learned_addr learned my addr 172.21.15.150:0/1032490144 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 -- 172.21.15.150:0/1032490144 >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 msgr2=0x7fee780b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 --2- 172.21.15.150:0/1032490144 >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 0x7fee780b5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 -- 172.21.15.150:0/1032490144 >> v1:172.21.15.187:6789/0 conn(0x7fee780b5a60 legacy=0x7fee78072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7d27a640 1 -- 172.21.15.150:0/1032490144 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2292548931 0 0) 0x7fee70001110 con 0x7fee780b5a60 2024-04-25T14:03:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.039+0000 7fee7e27c640 1 -- 172.21.15.150:0/1032490144 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee780ceec0 con 0x7fee78161bd0 2024-04-25T14:03:41.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.040+0000 7fee7da7b640 1 --2- 172.21.15.150:0/1032490144 >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 0x7fee780b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:03:41.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.040+0000 7fee7e27c640 1 --2- 172.21.15.150:0/1032490144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee74009e50 tx=0x7fee7402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52557b130ab1a78b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.040+0000 7fee7d27a640 1 -- 172.21.15.150:0/1032490144 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74036070 con 0x7fee78161bd0 2024-04-25T14:03:41.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.040+0000 7fee7d27a640 1 -- 172.21.15.150:0/1032490144 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee74031600 con 0x7fee78161bd0 2024-04-25T14:03:41.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.040+0000 7fee7d27a640 1 -- 172.21.15.150:0/1032490144 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee740319d0 con 0x7fee78161bd0 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 -- 172.21.15.150:0/1032490144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 msgr2=0x7fee7815e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 --2- 172.21.15.150:0/1032490144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee74009e50 tx=0x7fee7402a680 comp rx=0 tx=0).stop 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 -- 172.21.15.150:0/1032490144 shutdown_connections 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 --2- 172.21.15.150:0/1032490144 >> v2:172.21.15.187:3300/0 conn(0x7fee780ce390 0x7fee780b5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 --2- 172.21.15.150:0/1032490144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee78161bd0 0x7fee7815e050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 -- 172.21.15.150:0/1032490144 >> 172.21.15.150:0/1032490144 conn(0x7fee780d0520 msgr2=0x7fee780d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 -- 172.21.15.150:0/1032490144 shutdown_connections 2024-04-25T14:03:41.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.041+0000 7fee804e0640 1 -- 172.21.15.150:0/1032490144 wait complete. 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.043+0000 7fee804e0640 1 Processor -- start 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.043+0000 7fee804e0640 1 -- start start 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee780b5a60 0x7fee780b1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee780ce390 0x7fee780ab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee780d3800 con 0x7fee78161bd0 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee780d22d0 con 0x7fee780b5a60 2024-04-25T14:03:41.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee804e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee780b6d80 con 0x7fee780ce390 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee7ea7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee7ea7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49778/0 (socket says 172.21.15.150:49778) 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2453048328 learned_addr learned my addr 172.21.15.150:0/2453048328 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee7da7b640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee780ce390 0x7fee780ab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.044+0000 7fee7e27c640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee780b5a60 0x7fee780b1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee780ce390 msgr2=0x7fee780ab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee780ce390 0x7fee780ab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee780b5a60 msgr2=0x7fee780b1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee780b5a60 0x7fee780b1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2453048328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee780ac310 con 0x7fee78161bd0 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee7000c410 tx=0x7fee7000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.046 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee6effd640 1 -- 172.21.15.150:0/2453048328 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70017070 con 0x7fee78161bd0 2024-04-25T14:03:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee6effd640 1 -- 172.21.15.150:0/2453048328 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee700035f0 con 0x7fee78161bd0 2024-04-25T14:03:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee780ac5f0 con 0x7fee78161bd0 2024-04-25T14:03:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee6effd640 1 -- 172.21.15.150:0/2453048328 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee700039e0 con 0x7fee78161bd0 2024-04-25T14:03:41.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.045+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee780acbb0 con 0x7fee78161bd0 2024-04-25T14:03:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.047+0000 7fee6effd640 1 -- 172.21.15.150:0/2453048328 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee7001a070 con 0x7fee78161bd0 2024-04-25T14:03:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.047+0000 7fee6effd640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 0x7fee5c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.047+0000 7fee6effd640 1 -- 172.21.15.150:0/2453048328 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7fee700946b0 con 0x7fee78161bd0 2024-04-25T14:03:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.047+0000 7fee7e27c640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 0x7fee5c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.048+0000 7fee7e27c640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 0x7fee5c0769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fee74033000 tx=0x7fee74009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.048+0000 7fee87391740 1 Processor -- start 2024-04-25T14:03:41.049 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.049+0000 7fee87391740 1 -- start start 2024-04-25T14:03:41.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697ae270 0x564f69b7a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697853b0 0x564f69b76fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564f69ab4740 con 0x564f6978d590 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x564f69a503a0 con 0x564f697ae270 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x564f69ab0700 con 0x564f697853b0 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49784/0 (socket says 172.21.15.150:49784) 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2275708178 learned_addr learned my addr 172.21.15.150:0/2275708178 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7da7b640 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697853b0 0x564f69b76fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7e27c640 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697ae270 0x564f69b7a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697853b0 msgr2=0x564f69b76fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697853b0 0x564f69b76fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697ae270 msgr2=0x564f69b7a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697ae270 0x564f69b7a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 -- 172.21.15.150:0/2275708178 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f69a6c750 con 0x564f6978d590 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.050+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee70092940 tx=0x7fee7005ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b008cffd65081d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee5bfff640 1 -- 172.21.15.150:0/2275708178 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70062980 con 0x564f6978d590 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee5bfff640 1 -- 172.21.15.150:0/2275708178 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee700673a0 con 0x564f6978d590 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee5bfff640 1 -- 172.21.15.150:0/2275708178 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee70003d80 con 0x564f6978d590 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 -- 172.21.15.150:0/2275708178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 msgr2=0x564f69b50260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fee70092940 tx=0x7fee7005ad50 comp rx=0 tx=0).stop 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 -- 172.21.15.150:0/2275708178 shutdown_connections 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6978d590 0x564f69b50260 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697853b0 0x564f69b76fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 --2- 172.21.15.150:0/2275708178 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697ae270 0x564f69b7a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.051+0000 7fee87391740 1 -- 172.21.15.150:0/2275708178 >> 172.21.15.150:0/2275708178 conn(0x564f697a23c0 msgr2=0x564f69b4f9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.052+0000 7fee87391740 1 -- 172.21.15.150:0/2275708178 shutdown_connections 2024-04-25T14:03:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.052+0000 7fee87391740 1 -- 172.21.15.150:0/2275708178 wait complete. 2024-04-25T14:03:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.052+0000 7fee87391740 1 -- start start 2024-04-25T14:03:41.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.052+0000 7fee87391740 1 Processor -- start 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6978d590 0x564f69b7cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697ae270 0x564f69b7d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564f69a3bfe0 con 0x564f697853b0 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x564f69b760e0 con 0x564f6978d590 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x564f69b7d5d0 con 0x564f697ae270 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee87391740 10 client.0 _ll_register_callbacks cb 0x564f69a52150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49792/0 (socket says 172.21.15.150:49792) 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 -- 172.21.15.150:0/3753594156 learned_addr learned my addr 172.21.15.150:0/3753594156 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697ae270 0x564f69b7d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6978d590 0x564f69b7cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 -- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697ae270 msgr2=0x564f69b7d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697ae270 0x564f69b7d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 -- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6978d590 msgr2=0x564f69b7cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6978d590 0x564f69b7cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.054+0000 7fee7e27c640 1 -- 172.21.15.150:0/3753594156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f697c62d0 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee74009f40 tx=0x7fee74004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c36ce6db37a5464 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7404c070 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7fee7403d460 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee700673a0 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee74057050 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 10 client.5628 fetch_fsmap learned FSMap version 24 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) -- 0x564f698906b0 con 0x564f697853b0 2024-04-25T14:03:41.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee700673a0 con 0x564f697853b0 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 24) ==== 3494+0+0 (secure 0 0 0) 0x7fee7405a020 con 0x564f697853b0 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 10 client.5628 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 10 client.5628 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 20 client.5628 populate_metadata read hostname 'smithi150' 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f6976ce40 0x564f69b75420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 4 client.5628 mds_command: new command op to 5374 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 --> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f69ac2350 con 0x564f6976ce40 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.055+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f6976ce40 0x564f69b75420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.056+0000 7fee597fa640 10 client.5628 ms_handle_connect on v2:172.21.15.187:6833/4048124689 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.056+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f6976ce40 0x564f69b75420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb39b5422673408e server_cookie=96658766eda53e96 in_seq=0 out_seq=0 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee597fa640 1 -- 172.21.15.150:0/3753594156 <== mds.0 v2:172.21.15.187:6833/4048124689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564f69ac2350 con 0x564f6976ce40 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee597fa640 10 client.5628 handle_command_reply: tid=0 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 client.5628 shutdown 2024-04-25T14:03:41.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 msgr2=0x564f69b951a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fee74009f40 tx=0x7fee74004ab0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 shutdown_connections 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f6976ce40 0x564f69b75420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee597fa640 0 client.5628 ms_handle_reset on v2:172.21.15.187:6833/4048124689 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f697ae270 0x564f69b7d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6978d590 0x564f69b7cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 --2- 172.21.15.150:0/3753594156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69b951a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 >> 172.21.15.150:0/3753594156 conn(0x564f697a23c0 msgr2=0x564f69b4ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 shutdown_connections 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 1 -- 172.21.15.150:0/3753594156 wait complete. 2024-04-25T14:03:41.062 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.057+0000 7fee87391740 20 client.5628 trim_cache size 0 max 16384 2024-04-25T14:03:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.076+0000 7fee87391740 1 Processor -- start 2024-04-25T14:03:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.076+0000 7fee87391740 1 -- start start 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697853b0 0x564f69b7c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69b7a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee782003e0 con 0x564f6976ce40 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee780ccf80 con 0x564f697853b0 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee781ffdc0 con 0x564f6978d590 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee7e27c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49804/0 (socket says 172.21.15.150:49804) 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.077+0000 7fee7e27c640 1 -- 172.21.15.150:0/1216570855 learned_addr learned my addr 172.21.15.150:0/1216570855 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 -- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 msgr2=0x564f69b7a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69b7a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 -- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697853b0 msgr2=0x564f69b7c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7da7b640 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697853b0 0x564f69b7c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697853b0 0x564f69b7c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 -- 172.21.15.150:0/1216570855 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f69b4cb30 con 0x564f6976ce40 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee7e27c640 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fee74004740 tx=0x7fee74048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5eec503a5e961c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee5b7fe640 1 -- 172.21.15.150:0/1216570855 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee74060070 con 0x564f6976ce40 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee5b7fe640 1 -- 172.21.15.150:0/1216570855 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7403dc60 con 0x564f6976ce40 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.078+0000 7fee5b7fe640 1 -- 172.21.15.150:0/1216570855 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee7404c070 con 0x564f6976ce40 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 -- 172.21.15.150:0/1216570855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 msgr2=0x564f69bb7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fee74004740 tx=0x7fee74048000 comp rx=0 tx=0).stop 2024-04-25T14:03:41.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 -- 172.21.15.150:0/1216570855 shutdown_connections 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69b7a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f697853b0 0x564f69b7c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 --2- 172.21.15.150:0/1216570855 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f6976ce40 0x564f69bb7ea0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 -- 172.21.15.150:0/1216570855 >> 172.21.15.150:0/1216570855 conn(0x564f697a23c0 msgr2=0x564f69b76270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 -- 172.21.15.150:0/1216570855 shutdown_connections 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.079+0000 7fee87391740 1 -- 172.21.15.150:0/1216570855 wait complete. 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.080+0000 7fee87391740 1 -- start start 2024-04-25T14:03:41.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.080+0000 7fee87391740 1 Processor -- start 2024-04-25T14:03:41.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 0x564f69b7aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69ba6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x564f69b95820 con 0x564f697853b0 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x564f69b6fe90 con 0x564f6976ce40 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x564f69b934e0 con 0x564f6978d590 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee87391740 10 client.0 _ll_register_callbacks cb 0x7fee68009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee7da7b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee7da7b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:49812/0 (socket says 172.21.15.150:49812) 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.081+0000 7fee7da7b640 1 -- 172.21.15.150:0/3384702229 learned_addr learned my addr 172.21.15.150:0/3384702229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 0x564f69b7aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69ba6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 -- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 msgr2=0x564f69ba6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69ba6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 -- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 msgr2=0x564f69b7aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 0x564f69b7aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 -- 172.21.15.150:0/3384702229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f69ba39c0 con 0x564f697853b0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 0x564f69b7aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7ea7d640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69ba6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee7da7b640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fee780d4520 tx=0x7fee68004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95b417d26ea78d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee68003d80 con 0x564f697853b0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6273+0+0 (secure 0 0 0) 0x7fee68023d00 con 0x564f697853b0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee7403dc60 con 0x564f697853b0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fee68016430 con 0x564f697853b0 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 10 client.5634 fetch_fsmap learned FSMap version 24 2024-04-25T14:03:41.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) -- 0x564f69b4a260 con 0x564f697853b0 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee7403dc60 con 0x564f697853b0 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 24) ==== 3494+0+0 (secure 0 0 0) 0x7fee6802a050 con 0x564f697853b0 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 10 client.5634 fetch_fsmap finished waiting for FSMap version 24 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 10 client.5634 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 20 client.5634 populate_metadata read hostname 'smithi150' 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f69742280 0x564f69b96ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 4 client.5634 mds_command: new command op to 5374 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.082+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 --> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] -- command(tid 0: {"prefix": "status"}) -- 0x564f69b4af60 con 0x564f69742280 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f69742280 0x564f69b96ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee7e27c640 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f69742280 0x564f69b96ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94957ddcf2a0dcd6 server_cookie=21d0cd151f4dad0e in_seq=0 out_seq=0 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee597fa640 10 client.5634 ms_handle_connect on v2:172.21.15.187:6833/4048124689 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee597fa640 1 -- 172.21.15.150:0/3384702229 <== mds.0 v2:172.21.15.187:6833/4048124689 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x564f69b4af60 con 0x564f69742280 2024-04-25T14:03:41.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee597fa640 10 client.5634 handle_command_reply: tid=0 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee87391740 1 client.5634 shutdown 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 msgr2=0x564f69ba60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.083+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fee780d4520 tx=0x7fee68004410 comp rx=0 tx=0).stop 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 shutdown_connections 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:6833/4048124689,v1:172.21.15.187:6835/4048124689] conn(0x564f69742280 0x564f69b96ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x564f6978d590 0x564f69ba6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x564f697853b0 0x564f69ba60e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee597fa640 0 client.5634 ms_handle_reset on v2:172.21.15.187:6833/4048124689 2024-04-25T14:03:41.090 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 --2- 172.21.15.150:0/3384702229 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x564f6976ce40 0x564f69b7aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 >> 172.21.15.150:0/3384702229 conn(0x564f697a23c0 msgr2=0x564f69b4bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 shutdown_connections 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 1 -- 172.21.15.150:0/3384702229 wait complete. 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee87391740 20 client.5634 trim_cache size 0 max 16384 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5374, 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:03:41.091 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 5.3961290560000004, 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 24, 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 42, 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 42, 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 6.137699392 2024-04-25T14:03:41.092 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 msgr2=0x7fee5c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 0x7fee5c0769c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fee74033000 tx=0x7fee74009000 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 msgr2=0x7fee780af6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fee7000c410 tx=0x7fee7000c8d0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 shutdown_connections 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee5c0744b0 0x7fee5c0769c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee78161bd0 0x7fee780af6c0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee780ce390 0x7fee780ab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 --2- 172.21.15.150:0/2453048328 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee780b5a60 0x7fee780b1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 >> 172.21.15.150:0/2453048328 conn(0x7fee780d0520 msgr2=0x7fee780b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 shutdown_connections 2024-04-25T14:03:41.093 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:03:41.084+0000 7fee804e0640 1 -- 172.21.15.150:0/2453048328 wait complete. 2024-04-25T14:03:41.095 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:41.095 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:41.110 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file0", '"'"'w'"'"') as f: 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:41.111 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:41.112 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:41.112 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:41.112 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:41.152 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file0 2024-04-25T14:03:41.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:41.167 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file0': No such file or directory 2024-04-25T14:03:41.221 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:42.168 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file0 2024-04-25T14:03:42.187 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:42.187 INFO:tasks.cephfs.mount:File open_file0 became visible with size 7 from 0 after 1s 2024-04-25T14:03:42.188 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:42.188 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:42.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:42.246 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file0 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:42.316 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:42.316 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:42.317 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0086203 s, 122 MB/s 2024-04-25T14:03:42.317 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:42.317 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:42.374 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:42.374 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file1", '"'"'w'"'"') as f: 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:42.375 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:42.376 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:42.376 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:42.376 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:42.417 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file1 2024-04-25T14:03:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:42.434 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file1': No such file or directory 2024-04-25T14:03:42.487 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:43.223 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:43.435 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file1 2024-04-25T14:03:43.454 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:43.455 INFO:tasks.cephfs.mount:File open_file1 became visible with size 7 from 0 after 1s 2024-04-25T14:03:43.455 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:43.455 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:43.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:43.513 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file1 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:43.587 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:43.587 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:43.587 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00972702 s, 108 MB/s 2024-04-25T14:03:43.588 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:43.588 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:43.646 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:43.646 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:43.646 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:43.646 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file2", '"'"'w'"'"') as f: 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:43.647 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:43.689 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file2 2024-04-25T14:03:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:43.706 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file2': No such file or directory 2024-04-25T14:03:43.760 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:44.489 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:44.707 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file2 2024-04-25T14:03:44.726 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:44.726 INFO:tasks.cephfs.mount:File open_file2 became visible with size 7 from 0 after 1s 2024-04-25T14:03:44.726 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:44.726 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:44.785 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:44.785 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file2 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:44.859 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00984637 s, 106 MB/s 2024-04-25T14:03:44.859 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:44.860 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:44.917 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:44.917 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:44.917 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:44.917 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:44.917 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file3", '"'"'w'"'"') as f: 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:44.918 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:44.960 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file3 2024-04-25T14:03:44.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:44.976 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file3': No such file or directory 2024-04-25T14:03:45.028 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:45.225 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:45.763 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:45.978 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file3 2024-04-25T14:03:45.997 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:45.997 INFO:tasks.cephfs.mount:File open_file3 became visible with size 7 from 0 after 1s 2024-04-25T14:03:45.998 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:45.998 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:46.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:46.056 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file3 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:46.129 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:46.130 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:46.130 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970376 s, 108 MB/s 2024-04-25T14:03:46.130 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:46.130 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:46.187 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file4", '"'"'w'"'"') as f: 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:46.188 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:46.229 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file4 2024-04-25T14:03:46.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:46.246 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file4': No such file or directory 2024-04-25T14:03:46.298 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:46.491 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:47.030 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:47.228 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:47.247 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file4 2024-04-25T14:03:47.266 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:47.266 INFO:tasks.cephfs.mount:File open_file4 became visible with size 7 from 0 after 1s 2024-04-25T14:03:47.266 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:47.266 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:47.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:47.324 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file4 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:47.398 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:47.399 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:47.399 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00977441 s, 107 MB/s 2024-04-25T14:03:47.399 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:47.399 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:47.456 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:47.456 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file5", '"'"'w'"'"') as f: 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:47.457 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:47.458 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:47.458 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:47.500 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file5 2024-04-25T14:03:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:47.516 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file5': No such file or directory 2024-04-25T14:03:47.568 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:47.765 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:48.300 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:48.493 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:48.518 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file5 2024-04-25T14:03:48.536 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:48.536 INFO:tasks.cephfs.mount:File open_file5 became visible with size 7 from 0 after 1s 2024-04-25T14:03:48.536 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:48.536 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:48.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:48.594 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file5 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:48.665 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:48.665 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:48.666 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784146 s, 134 MB/s 2024-04-25T14:03:48.666 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:48.666 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file6", '"'"'w'"'"') as f: 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:48.723 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:48.724 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:48.766 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file6 2024-04-25T14:03:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:48.782 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file6': No such file or directory 2024-04-25T14:03:48.832 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:49.032 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:49.230 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:49.570 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:49.767 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:49.782 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file6 2024-04-25T14:03:49.801 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:49.801 INFO:tasks.cephfs.mount:File open_file6 became visible with size 7 from 0 after 1s 2024-04-25T14:03:49.801 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:49.802 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:49.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:49.859 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file6 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:49.932 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:49.932 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:49.932 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00832072 s, 126 MB/s 2024-04-25T14:03:49.933 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:49.933 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:49.990 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/open_file7", '"'"'w'"'"') as f: 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:03:49.991 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:03:50.033 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file7 2024-04-25T14:03:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:03:50.049 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/open_file7': No such file or directory 2024-04-25T14:03:50.101 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:50.302 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:50.834 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:51.034 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:51.049 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/open_file7 2024-04-25T14:03:51.068 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:03:51.068 INFO:tasks.cephfs.mount:File open_file7 became visible with size 7 from 0 after 1s 2024-04-25T14:03:51.068 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:03:51.069 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:03:51.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:03:51.127 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=open_file7 bs=1M conv=fdatasync count=1 seek=0) 2024-04-25T14:03:51.206 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records in 2024-04-25T14:03:51.206 INFO:teuthology.orchestra.run.smithi150.stderr:1+0 records out 2024-04-25T14:03:51.206 INFO:teuthology.orchestra.run.smithi150.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155758 s, 67.3 MB/s 2024-04-25T14:03:51.231 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:51.572 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:51.767 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:52.304 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:52.497 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:52.836 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:53.036 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:53.234 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:53.574 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:53.769 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:54.106 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:54.310 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:54.499 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:54.838 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:55.038 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:55.236 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:55.576 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:55.770 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:56.107 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:56.309 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:56.501 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:56.840 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:57.040 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:57.238 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:57.579 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:57.772 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:58.109 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:58.311 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:58.503 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:58.842 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:59.042 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:59.240 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:59.581 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:03:59.774 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:00.111 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:00.313 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:00.505 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:00.845 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:01.047 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:01.207 DEBUG:teuthology.orchestra.run.smithi150:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata stat mds0_openfiles.1 2024-04-25T14:04:01.242 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:01.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.268+0000 7f0bb29b3f40 1 Processor -- start 2024-04-25T14:04:01.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.268+0000 7f0bb29b3f40 1 -- start start 2024-04-25T14:04:01.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86ca64ae0 0x55c86ca7dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca5db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c86ca7fd40 con 0x55c86c9f48f0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55c86ca80290 con 0x55c86ca64ae0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55c86ca80570 con 0x55c86ca7f1e0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bac47b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bac47b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59576/0 (socket says 172.21.15.150:59576) 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.269+0000 7f0bac47b640 1 -- 172.21.15.150:0/1299526247 learned_addr learned my addr 172.21.15.150:0/1299526247 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bacc7c640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca5db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0babc7a640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86ca64ae0 0x55c86ca7dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 -- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 msgr2=0x55c86ca5db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca5db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 -- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86ca64ae0 msgr2=0x55c86ca7dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86ca64ae0 0x55c86ca7dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 -- 172.21.15.150:0/1299526247 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c86c8d3cb0 con 0x55c86c9f48f0 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bacc7c640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca5db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c054040 tx=0x7f0b9c050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec7c0a37d52b12b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0baac78640 1 -- 172.21.15.150:0/1299526247 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c055070 con 0x55c86c9f48f0 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0baac78640 1 -- 172.21.15.150:0/1299526247 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c051c20 con 0x55c86c9f48f0 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.270+0000 7f0baac78640 1 -- 172.21.15.150:0/1299526247 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c064440 con 0x55c86c9f48f0 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1299526247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 msgr2=0x55c86c9d68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c054040 tx=0x7f0b9c050000 comp rx=0 tx=0).stop 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1299526247 shutdown_connections 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca5db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86ca64ae0 0x55c86ca7dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1299526247 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86c9f48f0 0x55c86c9d68c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1299526247 >> 172.21.15.150:0/1299526247 conn(0x55c86c8ccac0 msgr2=0x55c86ca7d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1299526247 shutdown_connections 2024-04-25T14:04:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.271+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1299526247 wait complete. 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.272+0000 7f0bb29b3f40 1 Processor -- start 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.272+0000 7f0bb29b3f40 1 -- start start 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86c9f48f0 0x55c86ca76300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 0x55c86ca7da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca72570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55c86c9d5180 con 0x55c86ca64ae0 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55c86ca76860 con 0x55c86c9f48f0 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bb29b3f40 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55c86ca63bf0 con 0x55c86ca7f1e0 2024-04-25T14:04:01.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0babc7a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 0x55c86ca7da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0babc7a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 0x55c86ca7da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59584/0 (socket says 172.21.15.150:59584) 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0babc7a640 1 -- 172.21.15.150:0/1640439404 learned_addr learned my addr 172.21.15.150:0/1640439404 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86c9f48f0 0x55c86ca76300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0bacc7c640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca72570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0babc7a640 1 -- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 msgr2=0x55c86ca72570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.273+0000 7f0babc7a640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55c86ca7f1e0 0x55c86ca72570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0babc7a640 1 -- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86c9f48f0 msgr2=0x55c86ca76300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0babc7a640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55c86c9f48f0 0x55c86ca76300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0babc7a640 1 -- 172.21.15.150:0/1640439404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c86ca5a630 con 0x55c86ca64ae0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0babc7a640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 0x55c86ca7da20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009e50 tx=0x7f0ba0009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0ba9c76640 1 -- 172.21.15.150:0/1640439404 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba0016480 con 0x55c86ca64ae0 2024-04-25T14:04:01.277 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0ba9c76640 1 -- 172.21.15.150:0/1640439404 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ba0016620 con 0x55c86ca64ae0 2024-04-25T14:04:01.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c86ca5a910 con 0x55c86ca64ae0 2024-04-25T14:04:01.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0ba9c76640 1 -- 172.21.15.150:0/1640439404 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ba00169d0 con 0x55c86ca64ae0 2024-04-25T14:04:01.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.274+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c86c9aa760 con 0x55c86ca64ae0 2024-04-25T14:04:01.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0ba9c76640 1 -- 172.21.15.150:0/1640439404 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0ba0016b70 con 0x55c86ca64ae0 2024-04-25T14:04:01.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0ba9c76640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b98072e90 0x7f0b980753a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0ba9c76640 1 -- 172.21.15.150:0/1640439404 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f0ba00939a0 con 0x55c86ca64ae0 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b98072e90 0x7f0b980753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x55c86c9d5920 0x55c86c9aac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.276+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 --> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] -- osd_op(unknown.0.0:1 5.13 5:c9647462:::mds0_openfiles.1:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e44) -- 0x55c86c9ab1c0 con 0x55c86c9d5920 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bacc7c640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x55c86c9d5920 0x55c86c9aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bac47b640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b98072e90 0x7f0b980753a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c000f80 tx=0x7f0b9c051000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bacc7c640 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x55c86c9d5920 0x55c86c9aac80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:01.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bacc7c640 1 -- 172.21.15.150:0/1640439404 <== osd.3 v2:172.21.15.150:6800/690753092 1 ==== osd_op_reply(1 mds0_openfiles.1 [stat out=16b] v0'0 uv9 ondisk = 0) ==== 160+0+16 (crc 0 0 0) 0x7f0ba4009830 con 0x55c86c9d5920 2024-04-25T14:04:01.280 INFO:teuthology.orchestra.run.smithi150.stdout:cephfs_metadata/mds0_openfiles.1 mtime 2024-04-25T14:03:51.000000+0000, size 0 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x55c86c9d5920 msgr2=0x55c86c9aac80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x55c86c9d5920 0x55c86c9aac80 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b98072e90 msgr2=0x7f0b980753a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b98072e90 0x7f0b980753a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c000f80 tx=0x7f0b9c051000 comp rx=0 tx=0).stop 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 msgr2=0x55c86ca7da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:01.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.277+0000 7f0bb29b3f40 1 --2- 172.21.15.150:0/1640439404 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55c86ca64ae0 0x55c86ca7da20 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009e50 tx=0x7f0ba0009f40 comp rx=0 tx=0).stop 2024-04-25T14:04:01.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.278+0000 7f0bacc7c640 1 -- 172.21.15.150:0/1640439404 reap_dead start 2024-04-25T14:04:01.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.278+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 shutdown_connections 2024-04-25T14:04:01.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.278+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 >> 172.21.15.150:0/1640439404 conn(0x55c86c8ccac0 msgr2=0x55c86ca64120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:01.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.278+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 shutdown_connections 2024-04-25T14:04:01.282 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01.278+0000 7f0bb29b3f40 1 -- 172.21.15.150:0/1640439404 wait complete. 2024-04-25T14:04:01.282 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T14:04:01.282 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:04:01.361 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:04:01.403 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T14:04:01.403 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:04:01.404 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file0", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:01 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:01.583 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:01.776 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:02.113 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:02.315 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:02.507 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:02.845 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:03.046 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:03.258 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file1", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:03.299 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:03 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:03.585 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:03.777 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:04.115 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:04.317 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:04.509 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:04.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:04.522 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file2", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:04.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:04 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:04.847 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:05.048 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:05.587 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:05.779 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:05.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:05.792 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file3", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:05.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:05 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:06.118 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:06.318 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:06.849 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:07.049 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:07.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:07.060 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file4", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:07.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:07 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:07.589 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:08.120 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:08.320 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:08.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:08.334 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file5", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:08.376 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:08 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:08.791 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:04:08.795 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:04:08.851 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:09.591 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:09.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:09.605 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file6", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:09.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:09 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:10.122 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:10.853 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:10.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:10.867 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/open_file7", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:04:10.909 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:10 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:04:12.124 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:04:12.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:12.137 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:12.137 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:12.195 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:12.196 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:12.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:12.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.414+0000 7fbd9319e640 1 Processor -- start 2024-04-25T14:04:12.414 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.414+0000 7fbd9319e640 1 -- start start 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd9319e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd9319e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c06cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd9319e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd8c06d590 con 0x7fbd8c06d180 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd9319e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbd8c0d3140 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd9319e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b3570 con 0x7fbd8c0b26d0 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd90f3a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.416 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd90f3a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50834/0 (socket says 172.21.15.150:50834) 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd90f3a640 1 -- 172.21.15.150:0/1712675762 learned_addr learned my addr 172.21.15.150:0/1712675762 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.415+0000 7fbd8bfff640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c06cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd90f3a640 1 -- 172.21.15.150:0/1712675762 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 msgr2=0x7fbd8c06cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd90f3a640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c06cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd90f3a640 1 -- 172.21.15.150:0/1712675762 >> v1:172.21.15.187:6789/0 conn(0x7fbd8c06d180 legacy=0x7fbd8c161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd90f3a640 1 -- 172.21.15.150:0/1712675762 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c0b3810 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd8bfff640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c06cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:12.417 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd90f3a640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009870 tx=0x7fbd8002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c85c2a7b147a211 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd8b7fe640 1 -- 172.21.15.150:0/1712675762 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80037070 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.416+0000 7fbd8b7fe640 1 -- 172.21.15.150:0/1712675762 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd80032440 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd8b7fe640 1 -- 172.21.15.150:0/1712675762 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd80032810 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 -- 172.21.15.150:0/1712675762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 msgr2=0x7fbd8c0b3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbd80009870 tx=0x7fbd8002a180 comp rx=0 tx=0).stop 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 -- 172.21.15.150:0/1712675762 shutdown_connections 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c06cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 --2- 172.21.15.150:0/1712675762 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c0b3030 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.419 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 -- 172.21.15.150:0/1712675762 >> 172.21.15.150:0/1712675762 conn(0x7fbd8c073dc0 msgr2=0x7fbd8c0b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:12.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 -- 172.21.15.150:0/1712675762 shutdown_connections 2024-04-25T14:04:12.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.417+0000 7fbd9319e640 1 -- 172.21.15.150:0/1712675762 wait complete. 2024-04-25T14:04:12.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.419+0000 7fbd9319e640 1 Processor -- start 2024-04-25T14:04:12.420 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.419+0000 7fbd9319e640 1 -- start start 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.420+0000 7fbd9319e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c06d180 0x7fbd8c151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.420+0000 7fbd9319e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c0b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.420+0000 7fbd9319e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.420+0000 7fbd9319e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b21b0 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9319e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b1ed0 con 0x7fbd8c06d180 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9319e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8c0b3af0 con 0x7fbd8c0b26d0 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50840/0 (socket says 172.21.15.150:50840) 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 -- 172.21.15.150:0/822828847 learned_addr learned my addr 172.21.15.150:0/822828847 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:12.422 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd8bfff640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c0b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd90f3a640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c06d180 0x7fbd8c151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 -- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 msgr2=0x7fbd8c0b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c0b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 -- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c06d180 msgr2=0x7fbd8c151190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c06d180 0x7fbd8c151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 -- 172.21.15.150:0/822828847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8c158820 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd8bfff640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9173b640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400be60 tx=0x7fbd8400bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84019070 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd84014440 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd8c158b00 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd84014870 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.423 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.421+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd8c1590c0 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.422+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd8c050740 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.423+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbd8401c070 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.423+0000 7fbd897fa640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 0x7fbd700769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.424 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.424+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7fbd84095460 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.424+0000 7fbd90f3a640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 0x7fbd700769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.425 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.424+0000 7fbd90f3a640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 0x7fbd700769c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd80034000 tx=0x7fbd8002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.428 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.428+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbd8405f9a0 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.608+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbd8c0b4cb0 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.610+0000 7fbd897fa640 1 -- 172.21.15.150:0/822828847 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17254 (secure 0 0 0) 0x7fbd8405f2a0 con 0x7fbd8c0d2d30 2024-04-25T14:04:12.610 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:12.610 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:50.737672+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6833/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.150:6836/3196344346":"2024-04-26T14:03:34.944594+0000","172.21.15.150:6834/3196344346":"2024-04-26T14:03:34.944594+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:12.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 msgr2=0x7fbd700769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 0x7fbd700769c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbd80034000 tx=0x7fbd8002e000 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 msgr2=0x7fbd8c162ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400be60 tx=0x7fbd8400bf50 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 shutdown_connections 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbd700744b0 0x7fbd700769c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbd8c0d2d30 0x7fbd8c162ae0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbd8c0b26d0 0x7fbd8c0b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 --2- 172.21.15.150:0/822828847 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbd8c06d180 0x7fbd8c151190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 >> 172.21.15.150:0/822828847 conn(0x7fbd8c073dc0 msgr2=0x7fbd8c068520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 shutdown_connections 2024-04-25T14:04:12.613 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.612+0000 7fbd9319e640 1 -- 172.21.15.150:0/822828847 wait complete. 2024-04-25T14:04:12.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:03:21.365773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:03:21.767122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:04:12.625 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:04:12.735 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.735+0000 7f3be7384640 1 Processor -- start 2024-04-25T14:04:12.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.735+0000 7f3be7384640 1 -- start start 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be7384640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be7384640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3be009b630 0x7f3be0169e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be7384640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3be006cad0 con 0x7f3be006c5c0 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be7384640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f3be006cdb0 con 0x7f3be009b630 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be7384640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3be006d090 con 0x7f3be00d23c0 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be491f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3be009b630 0x7f3be0169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be491f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:36962/0 (socket says 172.21.15.150:36962) 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f3be009b630 0x7f3be0169e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33434/0 (socket says 172.21.15.150:33434) 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be491f640 1 -- 172.21.15.150:0/818686898 learned_addr learned my addr 172.21.15.150:0/818686898 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:12.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5120640 1 --1- 172.21.15.150:0/818686898 >> v1:172.21.15.187:6789/0 conn(0x7f3be006c5c0 0x7f3be00d1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50514/0 (socket says 172.21.15.150:50514) 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3bd7fff640 1 -- 172.21.15.150:0/818686898 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3191498350 0 0) 0x7f3be006cad0 con 0x7f3be006c5c0 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3bd7fff640 1 -- 172.21.15.150:0/818686898 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3bcc003520 con 0x7f3be006c5c0 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 -- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 msgr2=0x7f3be006db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 -- 172.21.15.150:0/818686898 >> v1:172.21.15.187:6789/0 conn(0x7f3be006c5c0 legacy=0x7f3be00d1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be5921640 1 -- 172.21.15.150:0/818686898 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be006d370 con 0x7f3be009b630 2024-04-25T14:04:12.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.737+0000 7f3be491f640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.738+0000 7f3be5921640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be009b630 0x7f3be0169e70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80097a0 tx=0x7f3bd802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bac89886240bb72a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.738+0000 7f3bd7fff640 1 -- 172.21.15.150:0/818686898 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8037070 con 0x7f3be009b630 2024-04-25T14:04:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.738+0000 7f3bd7fff640 1 -- 172.21.15.150:0/818686898 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bd8032440 con 0x7f3be009b630 2024-04-25T14:04:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.738+0000 7f3bd7fff640 1 -- 172.21.15.150:0/818686898 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd80327f0 con 0x7f3be009b630 2024-04-25T14:04:12.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 -- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be009b630 msgr2=0x7f3be0169e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be009b630 0x7f3be0169e70 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3bd80097a0 tx=0x7f3bd802a180 comp rx=0 tx=0).stop 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 -- 172.21.15.150:0/818686898 shutdown_connections 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be009b630 0x7f3be0169e70 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 --2- 172.21.15.150:0/818686898 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be00d23c0 0x7f3be006db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 -- 172.21.15.150:0/818686898 >> 172.21.15.150:0/818686898 conn(0x7f3be00739a0 msgr2=0x7f3be00b6db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 -- 172.21.15.150:0/818686898 shutdown_connections 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.739+0000 7f3be7384640 1 -- 172.21.15.150:0/818686898 wait complete. 2024-04-25T14:04:12.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.741+0000 7f3be7384640 1 Processor -- start 2024-04-25T14:04:12.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.741+0000 7f3be7384640 1 -- start start 2024-04-25T14:04:12.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be006c5c0 0x7f3be01486d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be009b630 0x7f3be01f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3be0069050 con 0x7f3be01f6820 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3be0163b10 con 0x7f3be006c5c0 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be7384640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3be00691d0 con 0x7f3be009b630 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be5921640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be5921640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50854/0 (socket says 172.21.15.150:50854) 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be5921640 1 -- 172.21.15.150:0/1207134260 learned_addr learned my addr 172.21.15.150:0/1207134260 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be491f640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be009b630 0x7f3be01f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.742+0000 7f3be5120640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be006c5c0 0x7f3be01486d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 -- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be009b630 msgr2=0x7f3be01f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be009b630 0x7f3be01f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 -- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be006c5c0 msgr2=0x7f3be01486d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be006c5c0 0x7f3be01486d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 -- 172.21.15.150:0/1207134260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be015cf90 con 0x7f3be01f6820 2024-04-25T14:04:12.744 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be5921640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3bd802a1c0 tx=0x7f3bd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8037040 con 0x7f3be01f6820 2024-04-25T14:04:12.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3bd803b430 con 0x7f3be01f6820 2024-04-25T14:04:12.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd803b7e0 con 0x7f3be01f6820 2024-04-25T14:04:12.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be015d270 con 0x7f3be01f6820 2024-04-25T14:04:12.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.743+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be015d830 con 0x7f3be01f6820 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.744+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be01f6c60 con 0x7f3be01f6820 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.745+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3bd8044020 con 0x7f3be01f6820 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.746+0000 7f3bd5ffb640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 0x7f3bc40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.746+0000 7f3be5120640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 0x7f3bc40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.746+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f3bd80b5390 con 0x7f3be01f6820 2024-04-25T14:04:12.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.746+0000 7f3be5120640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 0x7f3bc40769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0012e0 tx=0x7f3bdc001630 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:12.750 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.750+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3bd807f310 con 0x7f3be01f6820 2024-04-25T14:04:12.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.926+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3be0075290 con 0x7f3be01f6820 2024-04-25T14:04:12.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.926+0000 7f3bd5ffb640 1 -- 172.21.15.150:0/1207134260 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f3be0075290 con 0x7f3be01f6820 2024-04-25T14:04:12.927 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.928+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 msgr2=0x7f3bc40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.928+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 0x7f3bc40769c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3bdc0012e0 tx=0x7f3bdc001630 comp rx=0 tx=0).stop 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.928+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 msgr2=0x7f3be015e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.928+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3bd802a1c0 tx=0x7f3bd80046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 shutdown_connections 2024-04-25T14:04:12.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3bc40744b0 0x7f3bc40769c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3be01f6820 0x7f3be015e030 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3be009b630 0x7f3be01f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 --2- 172.21.15.150:0/1207134260 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3be006c5c0 0x7f3be01486d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 >> 172.21.15.150:0/1207134260 conn(0x7f3be00739a0 msgr2=0x7f3be0155870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 shutdown_connections 2024-04-25T14:04:12.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:12.929+0000 7f3be7384640 1 -- 172.21.15.150:0/1207134260 wait complete. 2024-04-25T14:04:12.940 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:04:13.050 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.050+0000 7fb89beb6640 1 Processor -- start 2024-04-25T14:04:13.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.050+0000 7fb89beb6640 1 -- start start 2024-04-25T14:04:13.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89beb6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb8940cebe0 0x7fb8941648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89beb6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8940d1ae0 0x7fb8940d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.052 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89beb6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb894160810 con 0x7fb8940b45f0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89beb6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb8940d0300 con 0x7fb8940d1ae0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89beb6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb894164e30 con 0x7fb8940cebe0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89a453640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb8940d1ae0 0x7fb8940d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb899451640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb8940b45f0 0x7fb8940b4a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47888/0 (socket says 172.21.15.150:47888) 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb899451640 1 -- 172.21.15.150:0/3484089552 learned_addr learned my addr 172.21.15.150:0/3484089552 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb899c52640 1 --2- 172.21.15.150:0/3484089552 >> v2:172.21.15.187:3300/0 conn(0x7fb8940cebe0 0x7fb8941648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89a453640 1 -- 172.21.15.150:0/3484089552 >> v2:172.21.15.187:3300/0 conn(0x7fb8940cebe0 msgr2=0x7fb8941648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89a453640 1 --2- 172.21.15.150:0/3484089552 >> v2:172.21.15.187:3300/0 conn(0x7fb8940cebe0 0x7fb8941648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89a453640 1 -- 172.21.15.150:0/3484089552 >> v1:172.21.15.150:6789/0 conn(0x7fb8940b45f0 legacy=0x7fb8940b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb898c50640 1 -- 172.21.15.150:0/3484089552 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3248541743 0 0) 0x7fb894160810 con 0x7fb8940b45f0 2024-04-25T14:04:13.053 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.052+0000 7fb89a453640 1 -- 172.21.15.150:0/3484089552 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8940cf850 con 0x7fb8940d1ae0 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.053+0000 7fb89a453640 1 --2- 172.21.15.150:0/3484089552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940d1ae0 0x7fb8940d1ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb8940b8380 tx=0x7fb88c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884fd5ab416c9bcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.053+0000 7fb898c50640 1 -- 172.21.15.150:0/3484089552 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c007740 con 0x7fb8940d1ae0 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.053+0000 7fb898c50640 1 -- 172.21.15.150:0/3484089552 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb88c0078e0 con 0x7fb8940d1ae0 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.053+0000 7fb898c50640 1 -- 172.21.15.150:0/3484089552 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c007cb0 con 0x7fb8940d1ae0 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 -- 172.21.15.150:0/3484089552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940d1ae0 msgr2=0x7fb8940d1ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3484089552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940d1ae0 0x7fb8940d1ef0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fb8940b8380 tx=0x7fb88c02a680 comp rx=0 tx=0).stop 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 -- 172.21.15.150:0/3484089552 shutdown_connections 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3484089552 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940d1ae0 0x7fb8940d1ef0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3484089552 >> v2:172.21.15.187:3300/0 conn(0x7fb8940cebe0 0x7fb8941648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 -- 172.21.15.150:0/3484089552 >> 172.21.15.150:0/3484089552 conn(0x7fb8940b76d0 msgr2=0x7fb8940b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 -- 172.21.15.150:0/3484089552 shutdown_connections 2024-04-25T14:04:13.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.054+0000 7fb89beb6640 1 -- 172.21.15.150:0/3484089552 wait complete. 2024-04-25T14:04:13.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.056+0000 7fb89beb6640 1 Processor -- start 2024-04-25T14:04:13.056 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.056+0000 7fb89beb6640 1 -- start start 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 0x7fb8940c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8940d1ae0 0x7fb8940c6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940cfe10 con 0x7fb8940cebe0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb8940b4160 con 0x7fb8940b45f0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89beb6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8940b3df0 con 0x7fb8940d1ae0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50874/0 (socket says 172.21.15.150:50874) 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 -- 172.21.15.150:0/3114852873 learned_addr learned my addr 172.21.15.150:0/3114852873 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899c52640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 0x7fb8940c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb89a453640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8940d1ae0 0x7fb8940c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 -- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8940d1ae0 msgr2=0x7fb8940c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8940d1ae0 0x7fb8940c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 -- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 msgr2=0x7fb8940c8c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 0x7fb8940c8c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899451640 1 -- 172.21.15.150:0/3114852873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb894160e10 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.057+0000 7fb899c52640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 0x7fb8940c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb899451640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c460 tx=0x7fb88400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884018070 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb884007500 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8941610f0 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb884004a20 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.058+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8941616b0 con 0x7fb8940cebe0 2024-04-25T14:04:13.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.059+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb894050740 con 0x7fb8940cebe0 2024-04-25T14:04:13.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.059+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb88401b070 con 0x7fb8940cebe0 2024-04-25T14:04:13.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.060+0000 7fb88a7fc640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 0x7fb8780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.060+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7fb884094730 con 0x7fb8940cebe0 2024-04-25T14:04:13.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.060+0000 7fb899c52640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 0x7fb8780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.061 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.060+0000 7fb899c52640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 0x7fb8780769f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8940b1fc0 tx=0x7fb890008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.064+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb88405ec70 con 0x7fb8940cebe0 2024-04-25T14:04:13.240 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.240+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb8940cf590 con 0x7fb8940cebe0 2024-04-25T14:04:13.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.241+0000 7fb88a7fc640 1 -- 172.21.15.150:0/3114852873 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7fb88405e570 con 0x7fb8940cebe0 2024-04-25T14:04:13.241 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:04:13.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 msgr2=0x7fb8780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 0x7fb8780769f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fb8940b1fc0 tx=0x7fb890008040 comp rx=0 tx=0).stop 2024-04-25T14:04:13.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 msgr2=0x7fb8940c63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb88400c460 tx=0x7fb88400c920 comp rx=0 tx=0).stop 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 shutdown_connections 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8780744e0 0x7fb8780769f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8940d1ae0 0x7fb8940c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8940cebe0 0x7fb8940c63e0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 --2- 172.21.15.150:0/3114852873 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb8940b45f0 0x7fb8940c8c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 >> 172.21.15.150:0/3114852873 conn(0x7fb8940b76d0 msgr2=0x7fb8940b1820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 shutdown_connections 2024-04-25T14:04:13.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.243+0000 7fb89beb6640 1 -- 172.21.15.150:0/3114852873 wait complete. 2024-04-25T14:04:13.254 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:04:13.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.364+0000 7fb87950d640 1 Processor -- start 2024-04-25T14:04:13.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.364+0000 7fb87950d640 1 -- start start 2024-04-25T14:04:13.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb87950d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb87406d350 0x7fb874161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb87950d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8740b3810 con 0x7fb8740d2d30 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb87950d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8740b1bf0 con 0x7fb8740b26d0 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb87950d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb8740d3140 con 0x7fb87406d350 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb8737fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb87406d350 0x7fb874161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb872ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb8740d2d30 0x7fb8740b3030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47892/0 (socket says 172.21.15.150:47892) 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb872ffd640 1 -- 172.21.15.150:0/1700986310 learned_addr learned my addr 172.21.15.150:0/1700986310 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb8737fe640 1 -- 172.21.15.150:0/1700986310 >> v1:172.21.15.187:6789/0 conn(0x7fb8740b26d0 legacy=0x7fb87406cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb8737fe640 1 -- 172.21.15.150:0/1700986310 >> v1:172.21.15.150:6789/0 conn(0x7fb8740d2d30 legacy=0x7fb8740b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb871ffb640 1 -- 172.21.15.150:0/1700986310 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1087651589 0 0) 0x7fb8740b3810 con 0x7fb8740d2d30 2024-04-25T14:04:13.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.366+0000 7fb8737fe640 1 -- 172.21.15.150:0/1700986310 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8740b1ed0 con 0x7fb87406d350 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.367+0000 7fb8737fe640 1 --2- 172.21.15.150:0/1700986310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87406d350 0x7fb874161ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097a0 tx=0x7fb86c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba8d7f945532696 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.367+0000 7fb871ffb640 1 -- 172.21.15.150:0/1700986310 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c037070 con 0x7fb87406d350 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.367+0000 7fb871ffb640 1 -- 172.21.15.150:0/1700986310 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c02ae50 con 0x7fb87406d350 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.367+0000 7fb871ffb640 1 -- 172.21.15.150:0/1700986310 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c032660 con 0x7fb87406d350 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 -- 172.21.15.150:0/1700986310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87406d350 msgr2=0x7fb874161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 --2- 172.21.15.150:0/1700986310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87406d350 0x7fb874161ab0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0097a0 tx=0x7fb86c02a140 comp rx=0 tx=0).stop 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 -- 172.21.15.150:0/1700986310 shutdown_connections 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 --2- 172.21.15.150:0/1700986310 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb87406d350 0x7fb874161ab0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 -- 172.21.15.150:0/1700986310 >> 172.21.15.150:0/1700986310 conn(0x7fb874073dc0 msgr2=0x7fb8740b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 -- 172.21.15.150:0/1700986310 shutdown_connections 2024-04-25T14:04:13.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.368+0000 7fb87950d640 1 -- 172.21.15.150:0/1700986310 wait complete. 2024-04-25T14:04:13.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.370+0000 7fb87950d640 1 Processor -- start 2024-04-25T14:04:13.370 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.370+0000 7fb87950d640 1 -- start start 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8740b26d0 0x7fb8741f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb87414e1e0 0x7fb8740b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb8740b2ae0 con 0x7fb8740d2d30 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb8740b3af0 con 0x7fb87414e1e0 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb87950d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb8740b2c60 con 0x7fb8740b26d0 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb8727fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb8727fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50894/0 (socket says 172.21.15.150:50894) 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb8727fc640 1 -- 172.21.15.150:0/4282128533 learned_addr learned my addr 172.21.15.150:0/4282128533 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:13.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.371+0000 7fb8737fe640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb87414e1e0 0x7fb8740b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb872ffd640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8740b26d0 0x7fb8741f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 -- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8740b26d0 msgr2=0x7fb8741f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8740b26d0 0x7fb8741f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 -- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb87414e1e0 msgr2=0x7fb8740b4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb87414e1e0 0x7fb8740b4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 -- 172.21.15.150:0/4282128533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87414f140 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb8727fc640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b3c0 tx=0x7fb86800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868011830 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb868016070 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb868011c50 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87414e8a0 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.372+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87414ee60 con 0x7fb8740d2d30 2024-04-25T14:04:13.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.373+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb874050740 con 0x7fb8740d2d30 2024-04-25T14:04:13.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.374+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb86801b460 con 0x7fb8740d2d30 2024-04-25T14:04:13.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.374+0000 7fb85b7fe640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 0x7fb8600769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.374+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7fb868094e80 con 0x7fb8740d2d30 2024-04-25T14:04:13.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.374+0000 7fb872ffd640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 0x7fb8600769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.375+0000 7fb872ffd640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 0x7fb8600769f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb864007ed0 tx=0x7fb864007e60 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.379+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb868020020 con 0x7fb8740d2d30 2024-04-25T14:04:13.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.553+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb87414e630 con 0x7fb8740d2d30 2024-04-25T14:04:13.553 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.553+0000 7fb85b7fe640 1 -- 172.21.15.150:0/4282128533 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7fb86805f450 con 0x7fb8740d2d30 2024-04-25T14:04:13.554 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 msgr2=0x7fb8600769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 0x7fb8600769f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb864007ed0 tx=0x7fb864007e60 comp rx=0 tx=0).stop 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 msgr2=0x7fb8740b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fb86800b3c0 tx=0x7fb86800b880 comp rx=0 tx=0).stop 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 shutdown_connections 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb8600744e0 0x7fb8600769f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb87414e1e0 0x7fb8740b4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb8740d2d30 0x7fb8740b3e30 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 --2- 172.21.15.150:0/4282128533 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb8740b26d0 0x7fb8741f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.555+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 >> 172.21.15.150:0/4282128533 conn(0x7fb874073dc0 msgr2=0x7fb87415b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.556+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 shutdown_connections 2024-04-25T14:04:13.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.556+0000 7fb87950d640 1 -- 172.21.15.150:0/4282128533 wait complete. 2024-04-25T14:04:13.567 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:04:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.675+0000 7efccac6e640 1 Processor -- start 2024-04-25T14:04:13.676 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.675+0000 7efccac6e640 1 -- start start 2024-04-25T14:04:13.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efccac6e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcc40b2d10 0x7efcc40d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efccac6e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 0x7efcc406f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efccac6e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcc4164ce0 con 0x7efcc40b5520 2024-04-25T14:04:13.678 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efccac6e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7efcc4163f20 con 0x7efcc40b2d10 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efccac6e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcc4164200 con 0x7efcc406ec50 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc8a0a640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7efcc40b2d10 0x7efcc40d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc3fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efcc40b5520 0x7efcc40b5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47900/0 (socket says 172.21.15.150:47900) 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc3fff640 1 -- 172.21.15.150:0/1441751437 learned_addr learned my addr 172.21.15.150:0/1441751437 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc920b640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 0x7efcc406f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1441751437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 msgr2=0x7efcc406f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc8a0a640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 0x7efcc406f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1441751437 >> v1:172.21.15.150:6789/0 conn(0x7efcc40b5520 legacy=0x7efcc40b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc37fe640 1 -- 172.21.15.150:0/1441751437 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2174954674 0 0) 0x7efcc4164ce0 con 0x7efcc40b5520 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1441751437 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc41644e0 con 0x7efcc40b2d10 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.677+0000 7efcc920b640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 0x7efcc406f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.678+0000 7efcc8a0a640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc40b2d10 0x7efcc40d1670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efcb40097a0 tx=0x7efcb402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5636e9f905ecf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.678+0000 7efcc37fe640 1 -- 172.21.15.150:0/1441751437 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4037070 con 0x7efcc40b2d10 2024-04-25T14:04:13.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.678+0000 7efcc37fe640 1 -- 172.21.15.150:0/1441751437 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb402ae50 con 0x7efcc40b2d10 2024-04-25T14:04:13.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.678+0000 7efcc37fe640 1 -- 172.21.15.150:0/1441751437 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4032660 con 0x7efcc40b2d10 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 -- 172.21.15.150:0/1441751437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc40b2d10 msgr2=0x7efcc40d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc40b2d10 0x7efcc40d1670 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7efcb40097a0 tx=0x7efcb402a100 comp rx=0 tx=0).stop 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 -- 172.21.15.150:0/1441751437 shutdown_connections 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc406ec50 0x7efcc406f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 --2- 172.21.15.150:0/1441751437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc40b2d10 0x7efcc40d1670 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 -- 172.21.15.150:0/1441751437 >> 172.21.15.150:0/1441751437 conn(0x7efcc40d0520 msgr2=0x7efcc40d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 -- 172.21.15.150:0/1441751437 shutdown_connections 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.679+0000 7efccac6e640 1 -- 172.21.15.150:0/1441751437 wait complete. 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.680+0000 7efccac6e640 1 Processor -- start 2024-04-25T14:04:13.681 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.681+0000 7efccac6e640 1 -- start start 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 0x7efcc415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcc40b5520 0x7efcc40b8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efcc41604c0 con 0x7efcc406ec50 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efcc40d1bb0 con 0x7efcc40b5520 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efccac6e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efcc4160640 con 0x7efcc40b2d10 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50924/0 (socket says 172.21.15.150:50924) 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1132182214 learned_addr learned my addr 172.21.15.150:0/1132182214 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:13.684 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc3fff640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 0x7efcc415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc920b640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcc40b5520 0x7efcc40b8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 msgr2=0x7efcc415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 0x7efcc415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcc40b5520 msgr2=0x7efcc40b8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcc40b5520 0x7efcc40b8d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc8a0a640 1 -- 172.21.15.150:0/1132182214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcc40b69e0 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.682+0000 7efcc3fff640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 0x7efcc415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efcc8a0a640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efcb4035040 tx=0x7efcb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4037040 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcb4004030 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcc40b6ef0 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcb4004400 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.683+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcc40b74b0 con 0x7efcc406ec50 2024-04-25T14:04:13.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.684+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcc40cee10 con 0x7efcc406ec50 2024-04-25T14:04:13.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.685+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efcb4003610 con 0x7efcc406ec50 2024-04-25T14:04:13.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.685+0000 7efcc17fa640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 0x7efcb00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:13.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.685+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7efcb40b5f80 con 0x7efcc406ec50 2024-04-25T14:04:13.686 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.685+0000 7efcc3fff640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 0x7efcb00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:13.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.685+0000 7efcc3fff640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 0x7efcb00769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efcc40b8880 tx=0x7efcb8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:13.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.690+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efcc40cee10 con 0x7efcc406ec50 2024-04-25T14:04:13.882 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.882+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efcc40cfb80 con 0x7efcc406ec50 2024-04-25T14:04:13.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.883+0000 7efcc17fa640 1 -- 172.21.15.150:0/1132182214 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v44) ==== 80+0+248 (secure 0 0 0) 0x7efcc40cfb80 con 0x7efcc406ec50 2024-04-25T14:04:13.883 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.187:6835/1241446695 2024-04-26T14:03:35.677482+0000 2024-04-25T14:04:13.883 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.187:6833/1241446695 2024-04-26T14:03:35.677482+0000 2024-04-25T14:04:13.883 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6836/3196344346 2024-04-26T14:03:34.944594+0000 2024-04-25T14:04:13.883 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150:6834/3196344346 2024-04-26T14:03:34.944594+0000 2024-04-25T14:04:13.884 INFO:teuthology.orchestra.run.smithi150.stderr:listed 4 entries 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 msgr2=0x7efcb00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 0x7efcb00769c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efcc40b8880 tx=0x7efcb8009040 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 msgr2=0x7efcc40b64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7efcb4035040 tx=0x7efcb40046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 shutdown_connections 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efcb00744b0 0x7efcb00769c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efcc40b5520 0x7efcc40b8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efcc40b2d10 0x7efcc415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 --2- 172.21.15.150:0/1132182214 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efcc406ec50 0x7efcc40b64a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:13.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 >> 172.21.15.150:0/1132182214 conn(0x7efcc40d0520 msgr2=0x7efcc415cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:13.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 shutdown_connections 2024-04-25T14:04:13.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:13.885+0000 7efccac6e640 1 -- 172.21.15.150:0/1132182214 wait complete. 2024-04-25T14:04:13.896 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T14:04:13.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:13.896 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T14:04:13.960 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:04:13.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:13.961 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T14:04:13.979 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:04:13.979 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:04:13.979 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:13.997 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:04:13.997 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:04:13.998 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:13.998 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:14.034 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:04:14.034 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:04:14.034 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:04:14.034 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:14.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:04:14.055 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:14.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.181+0000 7f03baa42640 1 Processor -- start 2024-04-25T14:04:14.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.181+0000 7f03baa42640 1 -- start start 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03baa42640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 0x7f03b406f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03baa42640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03baa42640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03b40cf2b0 con 0x7f03b40cfd80 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03baa42640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f03b40cecf0 con 0x7f03b40d34b0 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03baa42640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f03b40cefd0 con 0x7f03b40d0190 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b3fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 0x7f03b406f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b8fdf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b8fdf640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:37004/0 (socket says 172.21.15.150:37004) 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b3fff640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 0x7f03b406f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33498/0 (socket says 172.21.15.150:33498) 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b37fe640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f03b40cfd80 0x7f03b40d1c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50526/0 (socket says 172.21.15.150:50526) 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.182+0000 7f03b8fdf640 1 -- 172.21.15.150:0/1514706334 learned_addr learned my addr 172.21.15.150:0/1514706334 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.183+0000 7f03b8fdf640 1 -- 172.21.15.150:0/1514706334 >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 msgr2=0x7f03b406f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:14.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.183+0000 7f03b8fdf640 1 --2- 172.21.15.150:0/1514706334 >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 0x7f03b406f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.183+0000 7f03b8fdf640 1 -- 172.21.15.150:0/1514706334 >> v1:172.21.15.187:6789/0 conn(0x7f03b40cfd80 legacy=0x7f03b40d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.183+0000 7f03b8fdf640 1 -- 172.21.15.150:0/1514706334 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b40cd170 con 0x7f03b40d0190 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.183+0000 7f03b8fdf640 1 --2- 172.21.15.150:0/1514706334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009e50 tx=0x7f03ac02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6b32187b9f3b608 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03b2ffd640 1 -- 172.21.15.150:0/1514706334 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac036070 con 0x7f03b40d0190 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03b2ffd640 1 -- 172.21.15.150:0/1514706334 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03ac031600 con 0x7f03b40d0190 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03b2ffd640 1 -- 172.21.15.150:0/1514706334 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0319d0 con 0x7f03b40d0190 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 -- 172.21.15.150:0/1514706334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 msgr2=0x7f03b40cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 --2- 172.21.15.150:0/1514706334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009e50 tx=0x7f03ac02a710 comp rx=0 tx=0).stop 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 -- 172.21.15.150:0/1514706334 shutdown_connections 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 --2- 172.21.15.150:0/1514706334 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40d0190 0x7f03b40cd970 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 --2- 172.21.15.150:0/1514706334 >> v2:172.21.15.187:3300/0 conn(0x7f03b40d34b0 0x7f03b406f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 -- 172.21.15.150:0/1514706334 >> 172.21.15.150:0/1514706334 conn(0x7f03b409b630 msgr2=0x7f03b409ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 -- 172.21.15.150:0/1514706334 shutdown_connections 2024-04-25T14:04:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.184+0000 7f03baa42640 1 -- 172.21.15.150:0/1514706334 wait complete. 2024-04-25T14:04:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.186+0000 7f03baa42640 1 Processor -- start 2024-04-25T14:04:14.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.186+0000 7f03baa42640 1 -- start start 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.187+0000 7f03baa42640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40cfd80 0x7f03b4151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03baa42640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03baa42640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 0x7f03b406dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03baa42640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f03b40b2130 con 0x7f03b40d0190 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03baa42640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f03b40b1fb0 con 0x7f03b40d34b0 2024-04-25T14:04:14.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03baa42640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f03b40d2310 con 0x7f03b40cfd80 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50934/0 (socket says 172.21.15.150:50934) 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 -- 172.21.15.150:0/3528724648 learned_addr learned my addr 172.21.15.150:0/3528724648 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b3fff640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40cfd80 0x7f03b4151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b8fdf640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 0x7f03b406dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 -- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40cfd80 msgr2=0x7f03b4151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40cfd80 0x7f03b4151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 -- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 msgr2=0x7f03b406dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 0x7f03b406dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 -- 172.21.15.150:0/3528724648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b40c8130 con 0x7f03b40d0190 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b8fdf640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 0x7f03b406dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b37fe640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c3c0 tx=0x7f03a800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.188+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a80074e0 con 0x7f03b40d0190 2024-04-25T14:04:14.190 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.189+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a8007680 con 0x7f03b40d0190 2024-04-25T14:04:14.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.189+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b40c8410 con 0x7f03b40d0190 2024-04-25T14:04:14.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.189+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a8004a00 con 0x7f03b40d0190 2024-04-25T14:04:14.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.189+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b40c89d0 con 0x7f03b40d0190 2024-04-25T14:04:14.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.189+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b4050740 con 0x7f03b40d0190 2024-04-25T14:04:14.191 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.190+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f03a801a020 con 0x7f03b40d0190 2024-04-25T14:04:14.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.191+0000 7f03b0ff9640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 0x7f03a00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.191+0000 7f03b3fff640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 0x7f03a00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.191+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f03a8094af0 con 0x7f03b40d0190 2024-04-25T14:04:14.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.191+0000 7f03b3fff640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 0x7f03a00769c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009a90 tx=0x7f03a4009110 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.195+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f03a805f0c0 con 0x7f03b40d0190 2024-04-25T14:04:14.408 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.408+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03b40c7e80 con 0x7f03b40d0190 2024-04-25T14:04:14.409 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.408+0000 7f03b0ff9640 1 -- 172.21.15.150:0/3528724648 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5194 (secure 0 0 0) 0x7f03a805e9c0 con 0x7f03b40d0190 2024-04-25T14:04:14.409 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:14.409 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:39.701192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-04-25T14:04:14.409 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.410+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 msgr2=0x7f03a00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.410+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 0x7f03a00769c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009a90 tx=0x7f03a4009110 comp rx=0 tx=0).stop 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.410+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 msgr2=0x7f03b40c6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.410+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f03a800c3c0 tx=0x7f03a800c880 comp rx=0 tx=0).stop 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 shutdown_connections 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f03a00744b0 0x7f03a00769c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f03b40d34b0 0x7f03b406dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.411 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f03b40d0190 0x7f03b40c6fc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 --2- 172.21.15.150:0/3528724648 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f03b40cfd80 0x7f03b4151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 >> 172.21.15.150:0/3528724648 conn(0x7f03b409b630 msgr2=0x7f03b40b22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:14.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 shutdown_connections 2024-04-25T14:04:14.412 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.411+0000 7f03baa42640 1 -- 172.21.15.150:0/3528724648 wait complete. 2024-04-25T14:04:14.422 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.530+0000 7f4412540640 1 Processor -- start 2024-04-25T14:04:14.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.530+0000 7f4412540640 1 -- start start 2024-04-25T14:04:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 0x7f440c0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c0b26d0 0x7f440c06cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f440c06d180 0x7f440c161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f440c0d3140 con 0x7f440c06d180 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f440c06d590 con 0x7f440c0d2d30 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4412540640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f440c0b3570 con 0x7f440c0b26d0 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4410add640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f440c06d180 0x7f440c161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4410add640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f440c06d180 0x7f440c161ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50936/0 (socket says 172.21.15.150:50936) 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f4410add640 1 -- 172.21.15.150:0/3601163911 learned_addr learned my addr 172.21.15.150:0/3601163911 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f440b7fe640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c0b26d0 0x7f440c06cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.531+0000 7f440bfff640 1 --2- 172.21.15.150:0/3601163911 >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 0x7f440c0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 -- 172.21.15.150:0/3601163911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c0b26d0 msgr2=0x7f440c06cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c0b26d0 0x7f440c06cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 -- 172.21.15.150:0/3601163911 >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 msgr2=0x7f440c0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 --2- 172.21.15.150:0/3601163911 >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 0x7f440c0b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 -- 172.21.15.150:0/3601163911 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0b3810 con 0x7f440c06d180 2024-04-25T14:04:14.533 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f440bfff640 1 --2- 172.21.15.150:0/3601163911 >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 0x7f440c0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f4410add640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c06d180 0x7f440c161ab0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f440c06f3d0 tx=0x7f440402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3681d8a6b366d07c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f440affd640 1 -- 172.21.15.150:0/3601163911 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404036070 con 0x7f440c06d180 2024-04-25T14:04:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f440affd640 1 -- 172.21.15.150:0/3601163911 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4404031600 con 0x7f440c06d180 2024-04-25T14:04:14.534 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.532+0000 7f440affd640 1 -- 172.21.15.150:0/3601163911 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44040319d0 con 0x7f440c06d180 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 -- 172.21.15.150:0/3601163911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c06d180 msgr2=0x7f440c161ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c06d180 0x7f440c161ab0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f440c06f3d0 tx=0x7f440402a610 comp rx=0 tx=0).stop 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 -- 172.21.15.150:0/3601163911 shutdown_connections 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c06d180 0x7f440c161ab0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 --2- 172.21.15.150:0/3601163911 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c0b26d0 0x7f440c06cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 --2- 172.21.15.150:0/3601163911 >> v2:172.21.15.187:3300/0 conn(0x7f440c0d2d30 0x7f440c0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 -- 172.21.15.150:0/3601163911 >> 172.21.15.150:0/3601163911 conn(0x7f440c073dc0 msgr2=0x7f440c0b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:14.535 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 -- 172.21.15.150:0/3601163911 shutdown_connections 2024-04-25T14:04:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.533+0000 7f4412540640 1 -- 172.21.15.150:0/3601163911 wait complete. 2024-04-25T14:04:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.535+0000 7f4412540640 1 Processor -- start 2024-04-25T14:04:14.536 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.535+0000 7f4412540640 1 -- start start 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c06d180 0x7f440c151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f440c0b26d0 0x7f440c1610b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0b21b0 con 0x7f440c0d2d30 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f440c0b1ed0 con 0x7f440c0b26d0 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.536+0000 7f4412540640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f440c0b3af0 con 0x7f440c06d180 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50948/0 (socket says 172.21.15.150:50948) 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 -- 172.21.15.150:0/1963231979 learned_addr learned my addr 172.21.15.150:0/1963231979 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f440bfff640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c06d180 0x7f440c151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.538 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f440b7fe640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f440c0b26d0 0x7f440c1610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 -- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c06d180 msgr2=0x7f440c151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c06d180 0x7f440c151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 -- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f440c0b26d0 msgr2=0x7f440c1610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f440c0b26d0 0x7f440c1610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 -- 172.21.15.150:0/1963231979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f440c0d0f30 con 0x7f440c0d2d30 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4410add640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4404009f40 tx=0x7f4404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404036050 con 0x7f440c0d2d30 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4404042070 con 0x7f440c0d2d30 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f440c0d1210 con 0x7f440c0d2d30 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4404009510 con 0x7f440c0d2d30 2024-04-25T14:04:14.539 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.537+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f440c152100 con 0x7f440c0d2d30 2024-04-25T14:04:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.538+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f440c0d0ca0 con 0x7f440c0d2d30 2024-04-25T14:04:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.539+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4404007bd0 con 0x7f440c0d2d30 2024-04-25T14:04:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.539+0000 7f4408ff9640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.540+0000 7f440bfff640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.540 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.540+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f44040b61c0 con 0x7f440c0d2d30 2024-04-25T14:04:14.541 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.540+0000 7f440bfff640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 0x7f43f80769c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00aac0 tx=0x7f43fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.544+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f44040800b0 con 0x7f440c0d2d30 2024-04-25T14:04:14.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.721+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f440c06cbb0 con 0x7f440c0d2d30 2024-04-25T14:04:14.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.722+0000 7f4408ff9640 1 -- 172.21.15.150:0/1963231979 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17254 (secure 0 0 0) 0x7f44040843b0 con 0x7f440c0d2d30 2024-04-25T14:04:14.723 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:14.723 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:50.737672+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6833/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.150:6836/3196344346":"2024-04-26T14:03:34.944594+0000","172.21.15.150:6834/3196344346":"2024-04-26T14:03:34.944594+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 msgr2=0x7f43f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 0x7f43f80769c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f43fc00aac0 tx=0x7f43fc008040 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 msgr2=0x7f440c162be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f4404009f40 tx=0x7f4404004a40 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 shutdown_connections 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43f80744b0 0x7f43f80769c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f440c0d2d30 0x7f440c162be0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f440c0b26d0 0x7f440c1610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 --2- 172.21.15.150:0/1963231979 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f440c06d180 0x7f440c151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 >> 172.21.15.150:0/1963231979 conn(0x7f440c073dc0 msgr2=0x7f440c070d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:14.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 shutdown_connections 2024-04-25T14:04:14.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.724+0000 7f4412540640 1 -- 172.21.15.150:0/1963231979 wait complete. 2024-04-25T14:04:14.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:03:21.365773+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '37', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:03:21.767122+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:04:14.737 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:04:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.849+0000 7f163c96c640 1 Processor -- start 2024-04-25T14:04:14.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.849+0000 7f163c96c640 1 -- start start 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163c96c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163c96c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1638073a10 0x7f1638073e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163c96c640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1638074360 con 0x7f163815ac80 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163c96c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f163815da10 con 0x7f1638073a10 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163c96c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16380b0bd0 con 0x7f16380b07c0 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1635d9b640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1638073a10 0x7f1638073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1638073a10 0x7f1638073e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33540/0 (socket says 172.21.15.150:33540) 2024-04-25T14:04:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1635d9b640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:37040/0 (socket says 172.21.15.150:37040) 2024-04-25T14:04:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163659c640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f163815ac80 0x7f16380b67e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50542/0 (socket says 172.21.15.150:50542) 2024-04-25T14:04:14.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 -- 172.21.15.150:0/163976851 learned_addr learned my addr 172.21.15.150:0/163976851 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 -- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 msgr2=0x7f163815dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 -- 172.21.15.150:0/163976851 >> v1:172.21.15.187:6789/0 conn(0x7f163815ac80 legacy=0x7f16380b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f163559a640 1 -- 172.21.15.150:0/163976851 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961108659 0 0) 0x7f1628001490 con 0x7f163815ac80 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1636d9d640 1 -- 172.21.15.150:0/163976851 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16380b0ea0 con 0x7f1638073a10 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.851+0000 7f1635d9b640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.852+0000 7f1636d9d640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1638073a10 0x7f1638073e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad605f96eec8a5ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.852+0000 7f163559a640 1 -- 172.21.15.150:0/163976851 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630037070 con 0x7f1638073a10 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.852+0000 7f163559a640 1 -- 172.21.15.150:0/163976851 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630032440 con 0x7f1638073a10 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.852+0000 7f163559a640 1 -- 172.21.15.150:0/163976851 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630032810 con 0x7f1638073a10 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 -- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1638073a10 msgr2=0x7f1638073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1638073a10 0x7f1638073e20 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f163002a270 comp rx=0 tx=0).stop 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 -- 172.21.15.150:0/163976851 shutdown_connections 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1638073a10 0x7f1638073e20 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 --2- 172.21.15.150:0/163976851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f16380b07c0 0x7f163815dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 -- 172.21.15.150:0/163976851 >> 172.21.15.150:0/163976851 conn(0x7f16380d1ee0 msgr2=0x7f1638073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 -- 172.21.15.150:0/163976851 shutdown_connections 2024-04-25T14:04:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.853+0000 7f163c96c640 1 -- 172.21.15.150:0/163976851 wait complete. 2024-04-25T14:04:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.855+0000 7f163c96c640 1 Processor -- start 2024-04-25T14:04:14.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.855+0000 7f163c96c640 1 -- start start 2024-04-25T14:04:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 0x7f16381e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 0x7f16381f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f163815e3e0 con 0x7f163815ac80 2024-04-25T14:04:14.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f163815d6a0 con 0x7f16380b07c0 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.856+0000 7f163c96c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f16380b4c30 con 0x7f1638073a10 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50960/0 (socket says 172.21.15.150:50960) 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 -- 172.21.15.150:0/1190469049 learned_addr learned my addr 172.21.15.150:0/1190469049 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f163659c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 0x7f16381e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1635d9b640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 0x7f16381f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 -- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 msgr2=0x7f16381e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 0x7f16381e38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 -- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 msgr2=0x7f16381f5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 0x7f16381f5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 -- 172.21.15.150:0/1190469049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16381f2f20 con 0x7f163815ac80 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f163659c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 0x7f16381e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1635d9b640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 0x7f16381f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:14.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1636d9d640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f16300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630037040 con 0x7f163815ac80 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1630004030 con 0x7f163815ac80 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1638143f80 con 0x7f163815ac80 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1630004420 con 0x7f163815ac80 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.857+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16381f2440 con 0x7f163815ac80 2024-04-25T14:04:14.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.858+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16381f6300 con 0x7f163815ac80 2024-04-25T14:04:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.859+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f163003a070 con 0x7f163815ac80 2024-04-25T14:04:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.859+0000 7f1626ffd640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 0x7f16180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.860+0000 7f163659c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 0x7f16180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.860+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f16300b59f0 con 0x7f163815ac80 2024-04-25T14:04:14.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.860+0000 7f163659c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 0x7f16180769c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16280012e0 tx=0x7f1628005b90 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:14.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:14.864+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1630041020 con 0x7f163815ac80 2024-04-25T14:04:15.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.038+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1638050740 con 0x7f163815ac80 2024-04-25T14:04:15.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.039+0000 7f1626ffd640 1 -- 172.21.15.150:0/1190469049 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f163007ff40 con 0x7f163815ac80 2024-04-25T14:04:15.039 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:04:15.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 msgr2=0x7f16180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 0x7f16180769c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f16280012e0 tx=0x7f1628005b90 comp rx=0 tx=0).stop 2024-04-25T14:04:15.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 msgr2=0x7f1638158cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f1630009870 tx=0x7f16300046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 shutdown_connections 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f16180744b0 0x7f16180769c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f163815ac80 0x7f1638158cf0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f16380b07c0 0x7f16381f5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 --2- 172.21.15.150:0/1190469049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1638073a10 0x7f16381e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 >> 172.21.15.150:0/1190469049 conn(0x7f16380d1ee0 msgr2=0x7f163806f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 shutdown_connections 2024-04-25T14:04:15.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.041+0000 7f163c96c640 1 -- 172.21.15.150:0/1190469049 wait complete. 2024-04-25T14:04:15.053 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:04:15.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.163+0000 7f50e104a640 1 Processor -- start 2024-04-25T14:04:15.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.163+0000 7f50e104a640 1 -- start start 2024-04-25T14:04:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.164+0000 7f50e104a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc1648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.164+0000 7f50e104a640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50dc0b45f0 0x7f50dc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.164+0000 7f50e104a640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50dc160810 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.164+0000 7f50e104a640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f50dc0d0300 con 0x7f50dc0b45f0 2024-04-25T14:04:15.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.164+0000 7f50e104a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50dc164e30 con 0x7f50dc0cebe0 2024-04-25T14:04:15.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50dc0b45f0 0x7f50dc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50dad9d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc1648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f50dc0b45f0 0x7f50dc0b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33554/0 (socket says 172.21.15.150:33554) 2024-04-25T14:04:15.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50db59e640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f50dc0d1ae0 0x7f50dc0d1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50558/0 (socket says 172.21.15.150:50558) 2024-04-25T14:04:15.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 -- 172.21.15.150:0/2270584601 learned_addr learned my addr 172.21.15.150:0/2270584601 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 -- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 msgr2=0x7f50dc1648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 --2- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc1648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 -- 172.21.15.150:0/2270584601 >> v1:172.21.15.187:6789/0 conn(0x7f50dc0d1ae0 legacy=0x7f50dc0d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.165+0000 7f50da59c640 1 -- 172.21.15.150:0/2270584601 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc0b4f40 con 0x7f50dc0b45f0 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.166+0000 7f50da59c640 1 --2- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc0b4a00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f50d00098a0 tx=0x7f50d002a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83ec99eab529d026 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.166+0000 7f50d9d9b640 1 -- 172.21.15.150:0/2270584601 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0037070 con 0x7f50dc0b45f0 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.166+0000 7f50d9d9b640 1 -- 172.21.15.150:0/2270584601 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d0032440 con 0x7f50dc0b45f0 2024-04-25T14:04:15.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.166+0000 7f50d9d9b640 1 -- 172.21.15.150:0/2270584601 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0032810 con 0x7f50dc0b45f0 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 -- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 msgr2=0x7f50dc0b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 --2- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc0b4a00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f50d00098a0 tx=0x7f50d002a290 comp rx=0 tx=0).stop 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 -- 172.21.15.150:0/2270584601 shutdown_connections 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 --2- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc0b4a00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 --2- 172.21.15.150:0/2270584601 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc1648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 -- 172.21.15.150:0/2270584601 >> 172.21.15.150:0/2270584601 conn(0x7f50dc0b76d0 msgr2=0x7f50dc0b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 -- 172.21.15.150:0/2270584601 shutdown_connections 2024-04-25T14:04:15.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.167+0000 7f50e104a640 1 -- 172.21.15.150:0/2270584601 wait complete. 2024-04-25T14:04:15.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.169+0000 7f50e104a640 1 Processor -- start 2024-04-25T14:04:15.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.169+0000 7f50e104a640 1 -- start start 2024-04-25T14:04:15.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc0c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0cfe10 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f50dc0b4160 con 0x7f50dc0b45f0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50e104a640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f50dc0cfb30 con 0x7f50dc0cebe0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50db59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50db59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50964/0 (socket says 172.21.15.150:50964) 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50db59e640 1 -- 172.21.15.150:0/1547151831 learned_addr learned my addr 172.21.15.150:0/1547151831 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50dad9d640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.170+0000 7f50da59c640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc0c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 -- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 msgr2=0x7f50dc0c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc0c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 -- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 msgr2=0x7f50dc152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 -- 172.21.15.150:0/1547151831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50dc0c93f0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50db59e640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f50d400c3c0 tx=0x7f50d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d40074e0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50dc160a80 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50d4007680 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d40049e0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.171+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50dc161040 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.172+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50dc163940 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.173+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f50d401a020 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.173+0000 7f50c37fe640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 0x7f50c80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.173+0000 7f50dad9d640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 0x7f50c80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.173+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f50d4094aa0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.174+0000 7f50dad9d640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 0x7f50c80769c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0b1fc0 tx=0x7f50cc00b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.178+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f50d405e9a0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.350+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f50dc050740 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.351+0000 7f50c37fe640 1 -- 172.21.15.150:0/1547151831 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v44) ==== 104+0+11 (secure 0 0 0) 0x7f50d4062ca0 con 0x7f50dc0d1ae0 2024-04-25T14:04:15.351 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:04:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 msgr2=0x7f50c80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.353 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 0x7f50c80769c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f50dc0b1fc0 tx=0x7f50cc00b040 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 msgr2=0x7f50dc0b0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f50d400c3c0 tx=0x7f50d400c880 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 shutdown_connections 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f50c80744b0 0x7f50c80769c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f50dc0d1ae0 0x7f50dc0b0860 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f50dc0cebe0 0x7f50dc0c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 --2- 172.21.15.150:0/1547151831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f50dc0b45f0 0x7f50dc152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 >> 172.21.15.150:0/1547151831 conn(0x7f50dc0b76d0 msgr2=0x7f50dc0b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 shutdown_connections 2024-04-25T14:04:15.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.353+0000 7f50e104a640 1 -- 172.21.15.150:0/1547151831 wait complete. 2024-04-25T14:04:15.364 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:04:15.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.475+0000 7f9a989c5640 1 Processor -- start 2024-04-25T14:04:15.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.475+0000 7f9a989c5640 1 -- start start 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a989c5640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a940d4840 0x7f9a940d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a989c5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a9406c960 0x7f9a9406cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a989c5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a940d5190 con 0x7f9a94164710 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a989c5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9a94160940 con 0x7f9a9406c960 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a989c5640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9a94160c00 con 0x7f9a940d4840 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a92d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9a9406c960 0x7f9a9406cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a9259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9a94164710 0x7f9a94160230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47906/0 (socket says 172.21.15.150:47906) 2024-04-25T14:04:15.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.476+0000 7f9a9259c640 1 -- 172.21.15.150:0/2618886211 learned_addr learned my addr 172.21.15.150:0/2618886211 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a8bfff640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a940d4840 0x7f9a940d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a92d9d640 1 -- 172.21.15.150:0/2618886211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a940d4840 msgr2=0x7f9a940d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a92d9d640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a940d4840 0x7f9a940d4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a92d9d640 1 -- 172.21.15.150:0/2618886211 >> v1:172.21.15.150:6789/0 conn(0x7f9a94164710 legacy=0x7f9a94160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a92d9d640 1 -- 172.21.15.150:0/2618886211 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a94160ee0 con 0x7f9a9406c960 2024-04-25T14:04:15.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a91d9b640 1 -- 172.21.15.150:0/2618886211 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848109814 0 0) 0x7f9a84001110 con 0x7f9a94164710 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.477+0000 7f9a92d9d640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a9406c960 0x7f9a9406cd70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009d90 tx=0x7f9a8c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d8126809e2f369 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a91d9b640 1 -- 172.21.15.150:0/2618886211 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c007740 con 0x7f9a9406c960 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a91d9b640 1 -- 172.21.15.150:0/2618886211 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8c0078e0 con 0x7f9a9406c960 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a91d9b640 1 -- 172.21.15.150:0/2618886211 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8c007cb0 con 0x7f9a9406c960 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 -- 172.21.15.150:0/2618886211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a9406c960 msgr2=0x7f9a9406cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a9406c960 0x7f9a9406cd70 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c009d90 tx=0x7f9a8c02a660 comp rx=0 tx=0).stop 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 -- 172.21.15.150:0/2618886211 shutdown_connections 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a940d4840 0x7f9a940d4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 --2- 172.21.15.150:0/2618886211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a9406c960 0x7f9a9406cd70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 -- 172.21.15.150:0/2618886211 >> 172.21.15.150:0/2618886211 conn(0x7f9a940b77e0 msgr2=0x7f9a940b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 -- 172.21.15.150:0/2618886211 shutdown_connections 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.478+0000 7f9a989c5640 1 -- 172.21.15.150:0/2618886211 wait complete. 2024-04-25T14:04:15.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.480+0000 7f9a989c5640 1 Processor -- start 2024-04-25T14:04:15.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.480+0000 7f9a989c5640 1 -- start start 2024-04-25T14:04:15.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a9406c960 0x7f9a9414d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 0x7f9a940ba490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9a940d5630 con 0x7f9a940d4840 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9a941614a0 con 0x7f9a9406c960 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9a94164b20 con 0x7f9a94164710 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50982/0 (socket says 172.21.15.150:50982) 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 -- 172.21.15.150:0/1596813293 learned_addr learned my addr 172.21.15.150:0/1596813293 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a92d9d640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 0x7f9a940ba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a9259c640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a9406c960 0x7f9a9414d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 -- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 msgr2=0x7f9a940ba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 0x7f9a940ba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 -- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a9406c960 msgr2=0x7f9a9414d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a9406c960 0x7f9a9414d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 -- 172.21.15.150:0/1596813293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9414a330 con 0x7f9a940d4840 2024-04-25T14:04:15.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a92d9d640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 0x7f9a940ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8bfff640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00d620 tx=0x7f9a7c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00b620 con 0x7f9a940d4840 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a7c019070 con 0x7f9a940d4840 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a9414a7a0 con 0x7f9a940d4840 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.482+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00ba00 con 0x7f9a940d4840 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.483+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a9414ad60 con 0x7f9a940d4840 2024-04-25T14:04:15.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.483+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a94050740 con 0x7f9a940d4840 2024-04-25T14:04:15.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.484+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9a7c00bba0 con 0x7f9a940d4840 2024-04-25T14:04:15.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.485+0000 7f9a8affd640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 0x7f9a740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.485+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f9a7c0958b0 con 0x7f9a940d4840 2024-04-25T14:04:15.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.485+0000 7f9a9259c640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 0x7f9a740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.485+0000 7f9a9259c640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 0x7f9a740769f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f60 tx=0x7f9a84007c00 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.489+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9a7c05fdf0 con 0x7f9a940d4840 2024-04-25T14:04:15.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.663+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9a940d4110 con 0x7f9a940d4840 2024-04-25T14:04:15.664 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.664+0000 7f9a8affd640 1 -- 172.21.15.150:0/1596813293 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v44) ==== 100+0+11 (secure 0 0 0) 0x7f9a7c066e20 con 0x7f9a940d4840 2024-04-25T14:04:15.664 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:04:15.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 msgr2=0x7f9a740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 0x7f9a740769f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f60 tx=0x7f9a84007c00 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 msgr2=0x7f9a940c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c00d620 tx=0x7f9a7c00dae0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 shutdown_connections 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9a740744e0 0x7f9a740769f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9a94164710 0x7f9a940ba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9a940d4840 0x7f9a940c81b0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 --2- 172.21.15.150:0/1596813293 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9a9406c960 0x7f9a9414d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 >> 172.21.15.150:0/1596813293 conn(0x7f9a940b77e0 msgr2=0x7f9a940d0790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 shutdown_connections 2024-04-25T14:04:15.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.666+0000 7f9a989c5640 1 -- 172.21.15.150:0/1596813293 wait complete. 2024-04-25T14:04:15.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:15.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.785+0000 7f4403fd6640 1 Processor -- start 2024-04-25T14:04:15.785 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.785+0000 7f4403fd6640 1 -- start start 2024-04-25T14:04:15.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4403fd6640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f43fc0b2d10 0x7f43fc0d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4403fd6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc164ce0 con 0x7f43fc0b5520 2024-04-25T14:04:15.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4403fd6640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43fc163f20 con 0x7f43fc06ec50 2024-04-25T14:04:15.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4403fd6640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f43fc164200 con 0x7f43fc0b2d10 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401571640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f43fc0b5520 0x7f43fc0b5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47910/0 (socket says 172.21.15.150:47910) 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401571640 1 -- 172.21.15.150:0/3863855313 learned_addr learned my addr 172.21.15.150:0/3863855313 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401d72640 1 --2- 172.21.15.150:0/3863855313 >> v2:172.21.15.187:3300/0 conn(0x7f43fc0b2d10 0x7f43fc0d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2628785810 0 0) 0x7f43fc164ce0 con 0x7f43fc0b5520 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e8003520 con 0x7f43fc0b5520 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401d72640 1 -- 172.21.15.150:0/3863855313 >> v1:172.21.15.187:6789/0 conn(0x7f43fc06ec50 legacy=0x7f43fc06f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3406579945 0 0) 0x7f43e8003520 con 0x7f43fc0b5520 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401d72640 1 -- 172.21.15.150:0/3863855313 >> v1:172.21.15.150:6789/0 conn(0x7f43fc0b5520 legacy=0x7f43fc0b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.787+0000 7f4401d72640 1 -- 172.21.15.150:0/3863855313 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc1644e0 con 0x7f43fc0b2d10 2024-04-25T14:04:15.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.788+0000 7f4401d72640 1 --2- 172.21.15.150:0/3863855313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b2d10 0x7f43fc0d1670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097a0 tx=0x7f43f802a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76846b368525a311 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.788+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8037070 con 0x7f43fc0b2d10 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.788+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f802abc0 con 0x7f43fc0b2d10 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.788+0000 7f4400d70640 1 -- 172.21.15.150:0/3863855313 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ec001670 con 0x7f43fc0b2d10 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 -- 172.21.15.150:0/3863855313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b2d10 msgr2=0x7f43fc0d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3863855313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b2d10 0x7f43fc0d1670 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097a0 tx=0x7f43f802a130 comp rx=0 tx=0).stop 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 -- 172.21.15.150:0/3863855313 shutdown_connections 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3863855313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b2d10 0x7f43fc0d1670 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 -- 172.21.15.150:0/3863855313 >> 172.21.15.150:0/3863855313 conn(0x7f43fc0d0520 msgr2=0x7f43fc0d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 -- 172.21.15.150:0/3863855313 shutdown_connections 2024-04-25T14:04:15.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.789+0000 7f4403fd6640 1 -- 172.21.15.150:0/3863855313 wait complete. 2024-04-25T14:04:15.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.791+0000 7f4403fd6640 1 Processor -- start 2024-04-25T14:04:15.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.791+0000 7f4403fd6640 1 -- start start 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43fc0b2d10 0x7f43fc1e9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b5520 0x7f43fc1ea470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc1604c0 con 0x7f43fc06ec50 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43fc0d1bb0 con 0x7f43fc0b5520 2024-04-25T14:04:15.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4403fd6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43fc160640 con 0x7f43fc0b2d10 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4401d72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4401d72640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50988/0 (socket says 172.21.15.150:50988) 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4401d72640 1 -- 172.21.15.150:0/3590452367 learned_addr learned my addr 172.21.15.150:0/3590452367 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4401571640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43fc0b2d10 0x7f43fc1e9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.792+0000 7f4402573640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b5520 0x7f43fc1ea470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 -- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43fc0b2d10 msgr2=0x7f43fc1e9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43fc0b2d10 0x7f43fc1e9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 -- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b5520 msgr2=0x7f43fc1ea470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b5520 0x7f43fc1ea470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 -- 172.21.15.150:0/3590452367 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43fc158760 con 0x7f43fc06ec50 2024-04-25T14:04:15.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.793+0000 7f4401d72640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097a0 tx=0x7f43f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.795+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8041070 con 0x7f43fc06ec50 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.796+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f803c670 con 0x7f43fc06ec50 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.796+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f8037040 con 0x7f43fc06ec50 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.796+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43fc163820 con 0x7f43fc06ec50 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.796+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43fc163390 con 0x7f43fc06ec50 2024-04-25T14:04:15.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.797+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43fc050740 con 0x7f43fc06ec50 2024-04-25T14:04:15.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.797+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f43f8044070 con 0x7f43fc06ec50 2024-04-25T14:04:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.798+0000 7f43f27fc640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 0x7f43e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.798+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f43f80b5960 con 0x7f43fc06ec50 2024-04-25T14:04:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.798+0000 7f4401571640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 0x7f43e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:15.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.798+0000 7f4401571640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 0x7f43e00769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0698b0 tx=0x7f43ec000bc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:15.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:15.802+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43fc158760 con 0x7f43fc06ec50 2024-04-25T14:04:16.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.013+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f43fc06e520 con 0x7f43fc06ec50 2024-04-25T14:04:16.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.013+0000 7f43f27fc640 1 -- 172.21.15.150:0/3590452367 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5194 (secure 0 0 0) 0x7f43f807f8e0 con 0x7f43fc06ec50 2024-04-25T14:04:16.013 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:16.014 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:39.701192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-04-25T14:04:16.014 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 msgr2=0x7f43e00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 0x7f43e00769c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0698b0 tx=0x7f43ec000bc0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 msgr2=0x7f43fc1e99f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f43f80097a0 tx=0x7f43f80046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 shutdown_connections 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43e00744b0 0x7f43e00769c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43fc0b5520 0x7f43fc1ea470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43fc0b2d10 0x7f43fc1e9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 --2- 172.21.15.150:0/3590452367 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43fc06ec50 0x7f43fc1e99f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 >> 172.21.15.150:0/3590452367 conn(0x7f43fc0d0520 msgr2=0x7f43fc0b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.015+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 shutdown_connections 2024-04-25T14:04:16.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.016+0000 7f4403fd6640 1 -- 172.21.15.150:0/3590452367 wait complete. 2024-04-25T14:04:16.027 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:16.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.135+0000 7fac4d8e0640 1 Processor -- start 2024-04-25T14:04:16.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.135+0000 7fac4d8e0640 1 -- start start 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 0x7fac480d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac480b5520 0x7fac480b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480733c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fac480b5e70 con 0x7fac480b5520 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fac48164ce0 con 0x7fac480b2d10 2024-04-25T14:04:16.138 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac4d8e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac48163f20 con 0x7fac48072fb0 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac480b5520 0x7fac480b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fac480b5520 0x7fac480b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:50992/0 (socket says 172.21.15.150:50992) 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 -- 172.21.15.150:0/967021907 learned_addr learned my addr 172.21.15.150:0/967021907 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac46ffd640 1 --2- 172.21.15.150:0/967021907 >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 0x7fac480d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac477fe640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 -- 172.21.15.150:0/967021907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 msgr2=0x7fac480733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 -- 172.21.15.150:0/967021907 >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 msgr2=0x7fac480d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 --2- 172.21.15.150:0/967021907 >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 0x7fac480d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac467fc640 1 -- 172.21.15.150:0/967021907 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac48164200 con 0x7fac480b5520 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.137+0000 7fac46ffd640 1 --2- 172.21.15.150:0/967021907 >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 0x7fac480d1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.138+0000 7fac467fc640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac480b5930 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fac48073fe0 tx=0x7fac3c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd3b1a0871a319f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.138+0000 7fac45ffb640 1 -- 172.21.15.150:0/967021907 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c037070 con 0x7fac480b5520 2024-04-25T14:04:16.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.138+0000 7fac45ffb640 1 -- 172.21.15.150:0/967021907 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac3c032440 con 0x7fac480b5520 2024-04-25T14:04:16.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.138+0000 7fac45ffb640 1 -- 172.21.15.150:0/967021907 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac3c032810 con 0x7fac480b5520 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 -- 172.21.15.150:0/967021907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 msgr2=0x7fac480b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac480b5930 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fac48073fe0 tx=0x7fac3c02a140 comp rx=0 tx=0).stop 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 -- 172.21.15.150:0/967021907 shutdown_connections 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/967021907 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac480b5930 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/967021907 >> v2:172.21.15.187:3300/0 conn(0x7fac480b2d10 0x7fac480d1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 -- 172.21.15.150:0/967021907 >> 172.21.15.150:0/967021907 conn(0x7fac480d0520 msgr2=0x7fac480d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 -- 172.21.15.150:0/967021907 shutdown_connections 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.139+0000 7fac4d8e0640 1 -- 172.21.15.150:0/967021907 wait complete. 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.141+0000 7fac4d8e0640 1 Processor -- start 2024-04-25T14:04:16.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.141+0000 7fac4d8e0640 1 -- start start 2024-04-25T14:04:16.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480b7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480b2d10 0x7fac4815f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fac480d1bb0 con 0x7fac480b5520 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fac481647c0 con 0x7fac480b2d10 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac4d8e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fac481604c0 con 0x7fac48072fb0 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac477fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac477fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51006/0 (socket says 172.21.15.150:51006) 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.142+0000 7fac477fe640 1 -- 172.21.15.150:0/160542713 learned_addr learned my addr 172.21.15.150:0/160542713 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 -- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 msgr2=0x7fac480b7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac46ffd640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480b7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac467fc640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480b2d10 0x7fac4815f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480b7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 -- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480b2d10 msgr2=0x7fac4815f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480b2d10 0x7fac4815f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 -- 172.21.15.150:0/160542713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac480b91b0 con 0x7fac480b5520 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac477fe640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac4000d5d0 tx=0x7fac4000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40018020 con 0x7fac480b5520 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac40014620 con 0x7fac480b5520 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac481f9dc0 con 0x7fac480b5520 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac40014a10 con 0x7fac480b5520 2024-04-25T14:04:16.145 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.143+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac48162480 con 0x7fac480b5520 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.144+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac480b5340 con 0x7fac480b5520 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.145+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fac4001b020 con 0x7fac480b5520 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.145+0000 7fac2ffff640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 0x7fac340769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.145+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7fac40095630 con 0x7fac480b5520 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.145+0000 7fac46ffd640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 0x7fac340769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.146+0000 7fac46ffd640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 0x7fac340769f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ed0 tx=0x7fac3800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.150+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac480b5340 con 0x7fac480b5520 2024-04-25T14:04:16.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.325+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac48050740 con 0x7fac480b5520 2024-04-25T14:04:16.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.326+0000 7fac2ffff640 1 -- 172.21.15.150:0/160542713 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17254 (secure 0 0 0) 0x7fac40022030 con 0x7fac480b5520 2024-04-25T14:04:16.326 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:16.327 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:50.737672+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6833/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.150:6836/3196344346":"2024-04-26T14:03:34.944594+0000","172.21.15.150:6834/3196344346":"2024-04-26T14:03:34.944594+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 msgr2=0x7fac340769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 0x7fac340769f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fac38009ed0 tx=0x7fac3800b040 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 msgr2=0x7fac4815f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fac4000d5d0 tx=0x7fac4000da90 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 shutdown_connections 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fac340744e0 0x7fac340769f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fac480b5520 0x7fac4815f670 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fac480b2d10 0x7fac4815f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 --2- 172.21.15.150:0/160542713 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fac48072fb0 0x7fac480b7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 >> 172.21.15.150:0/160542713 conn(0x7fac480d0520 msgr2=0x7fac4806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 shutdown_connections 2024-04-25T14:04:16.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.328+0000 7fac4d8e0640 1 -- 172.21.15.150:0/160542713 wait complete. 2024-04-25T14:04:16.340 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T14:04:16.340 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:16.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.448+0000 7efe9bae7640 1 Processor -- start 2024-04-25T14:04:16.448 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.448+0000 7efe9bae7640 1 -- start start 2024-04-25T14:04:16.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe9bae7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efe94169100 0x7efe9416b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe9bae7640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe94164ce0 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe9bae7640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe94163f20 con 0x7efe940b2d10 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe9bae7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efe94164200 con 0x7efe94169100 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe99082640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efe940b5520 0x7efe940b5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47916/0 (socket says 172.21.15.150:47916) 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe99082640 1 -- 172.21.15.150:0/2240377191 learned_addr learned my addr 172.21.15.150:0/2240377191 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4105400178 0 0) 0x7efe94164ce0 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe9a084640 1 --2- 172.21.15.150:0/2240377191 >> v2:172.21.15.187:3300/0 conn(0x7efe94169100 0x7efe9416b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe80003520 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3836117479 0 0) 0x7efe80003520 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 >> v2:172.21.15.187:3300/0 conn(0x7efe94169100 msgr2=0x7efe9416b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 --2- 172.21.15.150:0/2240377191 >> v2:172.21.15.187:3300/0 conn(0x7efe94169100 0x7efe9416b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 >> v1:172.21.15.187:6789/0 conn(0x7efe940b2d10 legacy=0x7efe940d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe941644e0 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7efe84001670 con 0x7efe940b5520 2024-04-25T14:04:16.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 >> v1:172.21.15.150:6789/0 conn(0x7efe940b5520 legacy=0x7efe940b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe800039b0 0x7efe80003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80004800 0x7efe80004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 0x7efe8000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efe94164200 con 0x7efe80005250 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efe8000c1e0 con 0x7efe800039b0 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.450+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efe8000c3e0 con 0x7efe80004800 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 0x7efe8000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe99883640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe800039b0 0x7efe80003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe99082640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80004800 0x7efe80004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 -- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80004800 msgr2=0x7efe80004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80004800 0x7efe80004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 -- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe800039b0 msgr2=0x7efe80003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe800039b0 0x7efe80003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.454 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 -- 172.21.15.150:0/2240377191 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe80003520 con 0x7efe80005250 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 -- 172.21.15.150:0/2240377191 reap_dead start 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.451+0000 7efe9a084640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 0x7efe8000b9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009a90 tx=0x7efe80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbdb3227f8447ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c009690 con 0x7efe80005250 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c004020 con 0x7efe80005250 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe98881640 1 -- 172.21.15.150:0/2240377191 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c0043d0 con 0x7efe80005250 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 -- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 msgr2=0x7efe8000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 0x7efe8000b9c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7efe8c009a90 tx=0x7efe80005830 comp rx=0 tx=0).stop 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 -- 172.21.15.150:0/2240377191 shutdown_connections 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 --2- 172.21.15.150:0/2240377191 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe80005250 0x7efe8000b9c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 -- 172.21.15.150:0/2240377191 >> 172.21.15.150:0/2240377191 conn(0x7efe940d0520 msgr2=0x7efe940d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 -- 172.21.15.150:0/2240377191 shutdown_connections 2024-04-25T14:04:16.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.452+0000 7efe9bae7640 1 -- 172.21.15.150:0/2240377191 wait complete. 2024-04-25T14:04:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.454+0000 7efe9bae7640 1 Processor -- start 2024-04-25T14:04:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.454+0000 7efe9bae7640 1 -- start start 2024-04-25T14:04:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 0x7efe940b9190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 0x7efe940ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efe94160640 con 0x7efe940b8230 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efe941604c0 con 0x7efe940b64a0 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.455+0000 7efe9bae7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efe940b5930 con 0x7efe80005250 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51032/0 (socket says 172.21.15.150:51032) 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 -- 172.21.15.150:0/3576599628 learned_addr learned my addr 172.21.15.150:0/3576599628 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe99883640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 0x7efe940b9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe99082640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 0x7efe940ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 -- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 msgr2=0x7efe940b9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 0x7efe940b9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 -- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 msgr2=0x7efe940ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 0x7efe940ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 -- 172.21.15.150:0/3576599628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe940cd4e0 con 0x7efe940b8230 2024-04-25T14:04:16.459 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe99883640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 0x7efe940b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe99082640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 0x7efe940ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9a084640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efe8c02abb0 tx=0x7efe8c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c041070 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe8c003bd0 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe940b8b80 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe8c036040 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.456+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe941612d0 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.457+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe94050740 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.458+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efe8c044070 con 0x7efe940b8230 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.458+0000 7efe8a7fc640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 0x7efe780769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.458+0000 7efe99883640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 0x7efe780769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.460 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.458+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7efe8c0b4690 con 0x7efe940b8230 2024-04-25T14:04:16.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.459+0000 7efe99883640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 0x7efe780769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efe90000f80 tx=0x7efe90009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.463+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe940b8b80 con 0x7efe940b8230 2024-04-25T14:04:16.679 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.679+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe9406e520 con 0x7efe940b8230 2024-04-25T14:04:16.680 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.680+0000 7efe8a7fc640 1 -- 172.21.15.150:0/3576599628 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5194 (secure 0 0 0) 0x7efe940cd4e0 con 0x7efe940b8230 2024-04-25T14:04:16.680 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:16.680 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:39.701192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-04-25T14:04:16.681 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 msgr2=0x7efe780769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 0x7efe780769f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7efe90000f80 tx=0x7efe90009040 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 msgr2=0x7efe940b8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efe8c02abb0 tx=0x7efe8c004730 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 shutdown_connections 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efe780744e0 0x7efe780769f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efe940b8230 0x7efe940b8640 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efe940b64a0 0x7efe940ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 --2- 172.21.15.150:0/3576599628 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efe80005250 0x7efe940b9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 >> 172.21.15.150:0/3576599628 conn(0x7efe940d0520 msgr2=0x7efe94068520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 shutdown_connections 2024-04-25T14:04:16.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.682+0000 7efe9bae7640 1 -- 172.21.15.150:0/3576599628 wait complete. 2024-04-25T14:04:16.694 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:16.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.801+0000 7f29cdff8640 1 Processor -- start 2024-04-25T14:04:16.801 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.801+0000 7f29cdff8640 1 -- start start 2024-04-25T14:04:16.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29cdff8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29cdff8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29cdff8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c80b5330 con 0x7f29c80b3190 2024-04-25T14:04:16.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29cdff8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f29c80cfb70 con 0x7f29c80ce970 2024-04-25T14:04:16.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29cdff8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29c80d0250 con 0x7f29c80d2520 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c6ffd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:37156/0 (socket says 172.21.15.150:37156) 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c6ffd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:33626/0 (socket says 172.21.15.150:33626) 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 -- 172.21.15.150:0/705761891 learned_addr learned my addr 172.21.15.150:0/705761891 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c77fe640 1 --1- 172.21.15.150:0/705761891 >> v1:172.21.15.187:6789/0 conn(0x7f29c80b3190 0x7f29c80d1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:50580/0 (socket says 172.21.15.150:50580) 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 -- 172.21.15.150:0/705761891 >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 msgr2=0x7f29c80cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 --2- 172.21.15.150:0/705761891 >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 -- 172.21.15.150:0/705761891 >> v1:172.21.15.187:6789/0 conn(0x7f29c80b3190 legacy=0x7f29c80d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c7fff640 1 -- 172.21.15.150:0/705761891 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80cd1a0 con 0x7f29c80d2520 2024-04-25T14:04:16.804 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.803+0000 7f29c6ffd640 1 --2- 172.21.15.150:0/705761891 >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:16.805 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.804+0000 7f29c7fff640 1 --2- 172.21.15.150:0/705761891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c002a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=495c166a54180a7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.804+0000 7f29c67fc640 1 -- 172.21.15.150:0/705761891 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0036070 con 0x7f29c80d2520 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.804+0000 7f29c67fc640 1 -- 172.21.15.150:0/705761891 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c0031600 con 0x7f29c80d2520 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.804+0000 7f29c67fc640 1 -- 172.21.15.150:0/705761891 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c00319d0 con 0x7f29c80d2520 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 -- 172.21.15.150:0/705761891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 msgr2=0x7f29c80b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 --2- 172.21.15.150:0/705761891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c002a590 comp rx=0 tx=0).stop 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 -- 172.21.15.150:0/705761891 shutdown_connections 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 --2- 172.21.15.150:0/705761891 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80d2520 0x7f29c80b3850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 --2- 172.21.15.150:0/705761891 >> v2:172.21.15.187:3300/0 conn(0x7f29c80ce970 0x7f29c80cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 -- 172.21.15.150:0/705761891 >> 172.21.15.150:0/705761891 conn(0x7f29c809b630 msgr2=0x7f29c809ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 -- 172.21.15.150:0/705761891 shutdown_connections 2024-04-25T14:04:16.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.805+0000 7f29cdff8640 1 -- 172.21.15.150:0/705761891 wait complete. 2024-04-25T14:04:16.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.807+0000 7f29cdff8640 1 Processor -- start 2024-04-25T14:04:16.807 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.807+0000 7f29cdff8640 1 -- start start 2024-04-25T14:04:16.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c80b3190 0x7f29c8163630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80ce970 0x7f29c80ca4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80b5050 con 0x7f29c80d2520 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f29c80cd4d0 con 0x7f29c80b3190 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29cdff8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29c80b3d90 con 0x7f29c80ce970 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.808+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51042/0 (socket says 172.21.15.150:51042) 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 -- 172.21.15.150:0/470646242 learned_addr learned my addr 172.21.15.150:0/470646242 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c6ffd640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80ce970 0x7f29c80ca4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c77fe640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c80b3190 0x7f29c8163630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 -- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80ce970 msgr2=0x7f29c80ca4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80ce970 0x7f29c80ca4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 -- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c80b3190 msgr2=0x7f29c8163630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c80b3190 0x7f29c8163630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 -- 172.21.15.150:0/470646242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c80c9c50 con 0x7f29c80d2520 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29c7fff640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.810 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0036050 con 0x7f29c80d2520 2024-04-25T14:04:16.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29c0042070 con 0x7f29c80d2520 2024-04-25T14:04:16.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c80c9f30 con 0x7f29c80d2520 2024-04-25T14:04:16.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29c0009510 con 0x7f29c80d2520 2024-04-25T14:04:16.811 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.809+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c80caa00 con 0x7f29c80d2520 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.810+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c8050740 con 0x7f29c80d2520 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.811+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f29c0007b70 con 0x7f29c80d2520 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.811+0000 7f29affff640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 0x7f29b40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.812+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f29c00b60a0 con 0x7f29c80d2520 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.812+0000 7f29c77fe640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 0x7f29b40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:16.812 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.812+0000 7f29c77fe640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 0x7f29b40769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000f60 tx=0x7f29b8007e20 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:16.816 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:16.816+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29c00800b0 con 0x7f29c80d2520 2024-04-25T14:04:17.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.029+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29c81600f0 con 0x7f29c80d2520 2024-04-25T14:04:17.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.029+0000 7f29affff640 1 -- 172.21.15.150:0/470646242 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+5194 (secure 0 0 0) 0x7f29c00843b0 con 0x7f29c80d2520 2024-04-25T14:04:17.030 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:17.030 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":24,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:03:22.813096+0000","modified":"2024-04-25T14:03:39.701192+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5374},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5374":{"gid":5374,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":5,"addr":"172.21.15.187:6835/4048124689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":4048124689},{"type":"v1","addr":"172.21.15.187:6835","nonce":4048124689}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5374,"qdb_cluster":[5374]},"id":2}]} 2024-04-25T14:04:17.030 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 24 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 msgr2=0x7f29b40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 0x7f29b40769c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000f60 tx=0x7f29b8007e20 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 msgr2=0x7f29c80cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f29c0009e50 tx=0x7f29c0004a40 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 shutdown_connections 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.031+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f29b40744b0 0x7f29b40769c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29c80d2520 0x7f29c80cbfc0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f29c80ce970 0x7f29c80ca4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 --2- 172.21.15.150:0/470646242 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f29c80b3190 0x7f29c8163630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 >> 172.21.15.150:0/470646242 conn(0x7f29c809b630 msgr2=0x7f29c80d2930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 shutdown_connections 2024-04-25T14:04:17.033 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.032+0000 7f29cdff8640 1 -- 172.21.15.150:0/470646242 wait complete. 2024-04-25T14:04:17.043 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:17.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.151+0000 7fe6a0b34640 1 Processor -- start 2024-04-25T14:04:17.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.151+0000 7fe6a0b34640 1 -- start start 2024-04-25T14:04:17.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe6a0b34640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe69c0b5c90 0x7fe69c1602f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe6a0b34640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe69c0d1b90 0x7fe69c06ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe6a0b34640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe69c0ce710 con 0x7fe69c0b2d10 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe6a0b34640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe69c0b31f0 con 0x7fe69c0d1b90 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe6a0b34640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe69c164d10 con 0x7fe69c0b5c90 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe69c0d1b90 0x7fe69c06ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69a59c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe69c0b2d10 0x7fe69c0b5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47920/0 (socket says 172.21.15.150:47920) 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69a59c640 1 -- 172.21.15.150:0/658482486 learned_addr learned my addr 172.21.15.150:0/658482486 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe699d9b640 1 --2- 172.21.15.150:0/658482486 >> v2:172.21.15.187:3300/0 conn(0x7fe69c0b5c90 0x7fe69c1602f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69ad9d640 1 -- 172.21.15.150:0/658482486 >> v2:172.21.15.187:3300/0 conn(0x7fe69c0b5c90 msgr2=0x7fe69c1602f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69ad9d640 1 --2- 172.21.15.150:0/658482486 >> v2:172.21.15.187:3300/0 conn(0x7fe69c0b5c90 0x7fe69c1602f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69ad9d640 1 -- 172.21.15.150:0/658482486 >> v1:172.21.15.150:6789/0 conn(0x7fe69c0b2d10 legacy=0x7fe69c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69ad9d640 1 -- 172.21.15.150:0/658482486 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c163f50 con 0x7fe69c0d1b90 2024-04-25T14:04:17.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.153+0000 7fe69959a640 1 -- 172.21.15.150:0/658482486 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 374436431 0 0) 0x7fe68c001110 con 0x7fe69c0b2d10 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.154+0000 7fe69ad9d640 1 --2- 172.21.15.150:0/658482486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0d1b90 0x7fe69c06ed20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0cfcb0 tx=0x7fe69402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f00a1ab98e9c6e7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.154+0000 7fe69959a640 1 -- 172.21.15.150:0/658482486 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694007740 con 0x7fe69c0d1b90 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.154+0000 7fe69959a640 1 -- 172.21.15.150:0/658482486 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6940078e0 con 0x7fe69c0d1b90 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.154+0000 7fe69959a640 1 -- 172.21.15.150:0/658482486 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe694007cb0 con 0x7fe69c0d1b90 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 -- 172.21.15.150:0/658482486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0d1b90 msgr2=0x7fe69c06ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/658482486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0d1b90 0x7fe69c06ed20 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0cfcb0 tx=0x7fe69402a5f0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 -- 172.21.15.150:0/658482486 shutdown_connections 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/658482486 >> v2:172.21.15.187:3300/0 conn(0x7fe69c0b5c90 0x7fe69c1602f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/658482486 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0d1b90 0x7fe69c06ed20 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 -- 172.21.15.150:0/658482486 >> 172.21.15.150:0/658482486 conn(0x7fe69c0d0580 msgr2=0x7fe69c0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 -- 172.21.15.150:0/658482486 shutdown_connections 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.155+0000 7fe6a0b34640 1 -- 172.21.15.150:0/658482486 wait complete. 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.157+0000 7fe6a0b34640 1 Processor -- start 2024-04-25T14:04:17.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.157+0000 7fe6a0b34640 1 -- start start 2024-04-25T14:04:17.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe69c0d1b90 0x7fe69c1f7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe69c0b69c0 0x7fe69c0b6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c1647f0 con 0x7fe69c0b2d10 2024-04-25T14:04:17.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe69c164510 con 0x7fe69c0d1b90 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe6a0b34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe69c06e8b0 con 0x7fe69c0b69c0 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe69a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:51052/0 (socket says 172.21.15.150:51052) 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe69a59c640 1 -- 172.21.15.150:0/2633353382 learned_addr learned my addr 172.21.15.150:0/2633353382 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe69a59c640 1 -- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe69c0b69c0 msgr2=0x7fe69c0b6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.158+0000 7fe69a59c640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe69c0b69c0 0x7fe69c0b6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe69a59c640 1 -- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe69c0d1b90 msgr2=0x7fe69c1f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe69a59c640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe69c0d1b90 0x7fe69c1f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe69a59c640 1 -- 172.21.15.150:0/2633353382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe69c0b7310 con 0x7fe69c0b2d10 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe69a59c640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c460 tx=0x7fe68c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c018070 con 0x7fe69c0b2d10 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe68c007500 con 0x7fe69c0b2d10 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe69c1585b0 con 0x7fe69c0b2d10 2024-04-25T14:04:17.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68c004a20 con 0x7fe69c0b2d10 2024-04-25T14:04:17.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.159+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69c163620 con 0x7fe69c0b2d10 2024-04-25T14:04:17.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.160+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe69c15db30 con 0x7fe69c0b2d10 2024-04-25T14:04:17.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.161+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe68c01b070 con 0x7fe69c0b2d10 2024-04-25T14:04:17.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.161+0000 7fe68affd640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 0x7fe67c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.161+0000 7fe699d9b640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 0x7fe67c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.161+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7fe68c094710 con 0x7fe69c0b2d10 2024-04-25T14:04:17.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.162+0000 7fe699d9b640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 0x7fe67c0769c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe690006fd0 tx=0x7fe690008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.166+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe69c15db30 con 0x7fe69c0b2d10 2024-04-25T14:04:17.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.341+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe69c050740 con 0x7fe69c0b2d10 2024-04-25T14:04:17.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.342+0000 7fe68affd640 1 -- 172.21.15.150:0/2633353382 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+17254 (secure 0 0 0) 0x7fe69c0b7310 con 0x7fe69c0b2d10 2024-04-25T14:04:17.343 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:17.343 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":44,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:03:50.737672+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:03:21.365773+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-25T14:03:21.767122+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.30","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6833/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.150:6836/3196344346":"2024-04-26T14:03:34.944594+0000","172.21.15.150:6834/3196344346":"2024-04-26T14:03:34.944594+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 msgr2=0x7fe67c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 0x7fe67c0769c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe690006fd0 tx=0x7fe690008040 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 msgr2=0x7fe69c158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe68c00c460 tx=0x7fe68c00c920 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 shutdown_connections 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe67c0744b0 0x7fe67c0769c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe69c0b69c0 0x7fe69c0b6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe69c0d1b90 0x7fe69c1f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 --2- 172.21.15.150:0/2633353382 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe69c0b2d10 0x7fe69c158070 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 >> 172.21.15.150:0/2633353382 conn(0x7fe69c0d0580 msgr2=0x7fe69c15bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 shutdown_connections 2024-04-25T14:04:17.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.344+0000 7fe6a0b34640 1 -- 172.21.15.150:0/2633353382 wait complete. 2024-04-25T14:04:17.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T14:04:17.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.464+0000 7ff988415640 1 Processor -- start 2024-04-25T14:04:17.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.464+0000 7ff988415640 1 -- start start 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98015ac80 0x7ff9800b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9800b67e0 0x7ff98015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7ff980073a10 0x7ff980073e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7ff98015da10 con 0x7ff9800b67e0 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7ff980074360 con 0x7ff980073a10 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff988415640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9800b0ea0 con 0x7ff98015ac80 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.465+0000 7ff9859b0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9800b67e0 0x7ff98015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7ff9800b67e0 0x7ff98015dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34950/0 (socket says 172.21.15.150:34950) 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 -- 172.21.15.150:0/796739843 learned_addr learned my addr 172.21.15.150:0/796739843 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9869b2640 1 --2- 172.21.15.150:0/796739843 >> v2:172.21.15.187:3300/0 conn(0x7ff980073a10 0x7ff980073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9861b1640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98015ac80 0x7ff9800b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 -- 172.21.15.150:0/796739843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98015ac80 msgr2=0x7ff9800b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98015ac80 0x7ff9800b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 -- 172.21.15.150:0/796739843 >> v2:172.21.15.187:3300/0 conn(0x7ff980073a10 msgr2=0x7ff980073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 --2- 172.21.15.150:0/796739843 >> v2:172.21.15.187:3300/0 conn(0x7ff980073a10 0x7ff980073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.466+0000 7ff9859b0640 1 -- 172.21.15.150:0/796739843 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9800b1180 con 0x7ff9800b67e0 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff9859b0640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff98015dea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff980072d90 tx=0x7ff97002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221a6786de9c66a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff9851af640 1 -- 172.21.15.150:0/796739843 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970037070 con 0x7ff9800b67e0 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff9851af640 1 -- 172.21.15.150:0/796739843 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff970032440 con 0x7ff9800b67e0 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff9851af640 1 -- 172.21.15.150:0/796739843 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970032810 con 0x7ff9800b67e0 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff988415640 1 -- 172.21.15.150:0/796739843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 msgr2=0x7ff98015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.467+0000 7ff988415640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff98015dea0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff980072d90 tx=0x7ff97002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 -- 172.21.15.150:0/796739843 shutdown_connections 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 --2- 172.21.15.150:0/796739843 >> v2:172.21.15.187:3300/0 conn(0x7ff980073a10 0x7ff980073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff98015dea0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 --2- 172.21.15.150:0/796739843 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff98015ac80 0x7ff9800b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 -- 172.21.15.150:0/796739843 >> 172.21.15.150:0/796739843 conn(0x7ff9800d1ee0 msgr2=0x7ff980073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 -- 172.21.15.150:0/796739843 shutdown_connections 2024-04-25T14:04:17.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.468+0000 7ff988415640 1 -- 172.21.15.150:0/796739843 wait complete. 2024-04-25T14:04:17.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.469+0000 7ff988415640 1 Processor -- start 2024-04-25T14:04:17.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.469+0000 7ff988415640 1 -- start start 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 0x7ff98015cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff98015ac80 0x7ff980151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ff9800b6bf0 con 0x7ff9800b67e0 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ff98015e3e0 con 0x7ff98015ac80 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff988415640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ff9800b6d70 con 0x7ff980073a10 2024-04-25T14:04:17.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34956/0 (socket says 172.21.15.150:34956) 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 -- 172.21.15.150:0/2046049416 learned_addr learned my addr 172.21.15.150:0/2046049416 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9869b2640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff98015ac80 0x7ff980151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9861b1640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 0x7ff98015cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 -- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 msgr2=0x7ff98015cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 0x7ff98015cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 -- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff98015ac80 msgr2=0x7ff980151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff98015ac80 0x7ff980151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 -- 172.21.15.150:0/2046049416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98015d0b0 con 0x7ff9800b67e0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9861b1640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 0x7ff98015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.471+0000 7ff9859b0640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff97002a220 tx=0x7ff9700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.472+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970041020 con 0x7ff9800b67e0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.472+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff97003d620 con 0x7ff9800b67e0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.472+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff970037040 con 0x7ff9800b67e0 2024-04-25T14:04:17.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.472+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff980157d10 con 0x7ff9800b67e0 2024-04-25T14:04:17.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.472+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9801561f0 con 0x7ff9800b67e0 2024-04-25T14:04:17.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.473+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff980157a80 con 0x7ff9800b67e0 2024-04-25T14:04:17.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.473+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ff970044020 con 0x7ff9800b67e0 2024-04-25T14:04:17.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.474+0000 7ff976ffd640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 0x7ff9640769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.474+0000 7ff9861b1640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 0x7ff9640769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.474+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7ff9700b5a20 con 0x7ff9800b67e0 2024-04-25T14:04:17.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.474+0000 7ff9861b1640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 0x7ff9640769f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009a70 tx=0x7ff97c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.478+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff97004b030 con 0x7ff9800b67e0 2024-04-25T14:04:17.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.694+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff980050740 con 0x7ff9800b67e0 2024-04-25T14:04:17.695 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:04:17.694+0000 7f46d47d6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:04:17.696 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.696+0000 7ff976ffd640 1 -- 172.21.15.150:0/2046049416 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) ==== 161+0+0 (secure 0 0 0) 0x7ff97007fff0 con 0x7ff9800b67e0 2024-04-25T14:04:17.696 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 msgr2=0x7ff9640769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 0x7ff9640769f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff97c009a70 tx=0x7ff97c008040 comp rx=0 tx=0).stop 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 msgr2=0x7ff9801569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff97002a220 tx=0x7ff9700046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 shutdown_connections 2024-04-25T14:04:17.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ff9640744e0 0x7ff9640769f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ff98015ac80 0x7ff980151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.698+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ff9800b67e0 0x7ff9801569f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.699+0000 7ff988415640 1 --2- 172.21.15.150:0/2046049416 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ff980073a10 0x7ff98015cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.699+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 >> 172.21.15.150:0/2046049416 conn(0x7ff9800d1ee0 msgr2=0x7ff9800b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.699+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 shutdown_connections 2024-04-25T14:04:17.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.699+0000 7ff988415640 1 -- 172.21.15.150:0/2046049416 wait complete. 2024-04-25T14:04:17.704 INFO:tasks.ceph.mds.d.smithi187.stderr: -9610> 2024-04-25T14:03:40.688+0000 7f693e348640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:04:17.710 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T14:04:17.761 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:04:17.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.821+0000 7f95f76d8640 1 Processor -- start 2024-04-25T14:04:17.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.821+0000 7f95f76d8640 1 -- start start 2024-04-25T14:04:17.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f76d8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f95f00b5580 0x7f95f00b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f76d8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f00b5ed0 con 0x7f95f0163e60 2024-04-25T14:04:17.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f76d8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f0161820 con 0x7f95f0161410 2024-04-25T14:04:17.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f76d8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f95f00ce990 con 0x7f95f00b5580 2024-04-25T14:04:17.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f4c73640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f95f0163e60 0x7f95f015dcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45542/0 (socket says 172.21.15.150:45542) 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f4c73640 1 -- 172.21.15.150:0/799311965 learned_addr learned my addr 172.21.15.150:0/799311965 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.822+0000 7f95f5c75640 1 --2- 172.21.15.150:0/799311965 >> v2:172.21.15.187:3300/0 conn(0x7f95f00b5580 0x7f95f00b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2597204201 0 0) 0x7f95e0001110 con 0x7f95f0163e60 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95dc003520 con 0x7f95f0163e60 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2658427266 0 0) 0x7f95f0161820 con 0x7f95f0161410 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95e0001110 con 0x7f95f0161410 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95f5c75640 1 -- 172.21.15.150:0/799311965 >> v1:172.21.15.187:6789/0 conn(0x7f95f0161410 legacy=0x7f95f00d16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95f5c75640 1 -- 172.21.15.150:0/799311965 >> v1:172.21.15.150:6789/0 conn(0x7f95f0163e60 legacy=0x7f95f015dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 909028360 0 0) 0x7f95dc003520 con 0x7f95f0163e60 2024-04-25T14:04:17.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.823+0000 7f95f5c75640 1 -- 172.21.15.150:0/799311965 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f00ce660 con 0x7f95f00b5580 2024-04-25T14:04:17.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95f5c75640 1 --2- 172.21.15.150:0/799311965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f00b5580 0x7f95f00b5990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009d90 tx=0x7f95e802a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2d61fc1a428ed3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e0001840 con 0x7f95f00b5580 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95e8007540 con 0x7f95f00b5580 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95e7fff640 1 -- 172.21.15.150:0/799311965 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95e8007910 con 0x7f95f00b5580 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95f76d8640 1 -- 172.21.15.150:0/799311965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f00b5580 msgr2=0x7f95f00b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.824+0000 7f95f76d8640 1 --2- 172.21.15.150:0/799311965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f00b5580 0x7f95f00b5990 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009d90 tx=0x7f95e802a560 comp rx=0 tx=0).stop 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.825+0000 7f95f76d8640 1 -- 172.21.15.150:0/799311965 shutdown_connections 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.825+0000 7f95f76d8640 1 --2- 172.21.15.150:0/799311965 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f00b5580 0x7f95f00b5990 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.825+0000 7f95f76d8640 1 -- 172.21.15.150:0/799311965 >> 172.21.15.150:0/799311965 conn(0x7f95f00d0580 msgr2=0x7f95f00d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.825+0000 7f95f76d8640 1 -- 172.21.15.150:0/799311965 shutdown_connections 2024-04-25T14:04:17.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.825+0000 7f95f76d8640 1 -- 172.21.15.150:0/799311965 wait complete. 2024-04-25T14:04:17.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.826+0000 7f95f76d8640 1 Processor -- start 2024-04-25T14:04:17.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.826+0000 7f95f76d8640 1 -- start start 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 0x7f95f00b7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f0163e60 0x7f95f015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f95f0164bc0 con 0x7f95f00b5580 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f95f01648e0 con 0x7f95f0163e60 2024-04-25T14:04:17.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f95f0164270 con 0x7f95f0161410 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34962/0 (socket says 172.21.15.150:34962) 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 -- 172.21.15.150:0/579166929 learned_addr learned my addr 172.21.15.150:0/579166929 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5c75640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f0163e60 0x7f95f015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f4c73640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 0x7f95f00b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 -- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 msgr2=0x7f95f00b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 0x7f95f00b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 -- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f0163e60 msgr2=0x7f95f015bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f0163e60 0x7f95f015bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 -- 172.21.15.150:0/579166929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f00b8400 con 0x7f95f00b5580 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f4c73640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 0x7f95f00b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f5474640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c3c0 tx=0x7f95ec00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec017070 con 0x7f95f00b5580 2024-04-25T14:04:17.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ec0052f0 con 0x7f95f00b5580 2024-04-25T14:04:17.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f00b86e0 con 0x7f95f00b5580 2024-04-25T14:04:17.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.828+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec003830 con 0x7f95f00b5580 2024-04-25T14:04:17.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.829+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f00b8ca0 con 0x7f95f00b5580 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.830+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f0050740 con 0x7f95f00b5580 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.830+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f95ec01a070 con 0x7f95f00b5580 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.831+0000 7f95e5ffb640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 0x7f95d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.831+0000 7f95f4c73640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 0x7f95d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.831+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6388+0+0 (secure 0 0 0) 0x7f95ec094880 con 0x7f95f00b5580 2024-04-25T14:04:17.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.831+0000 7f95f4c73640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 0x7f95d40769c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f95e0001eb0 tx=0x7f95e0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:17.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:17.835+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f95ec05e7d0 con 0x7f95f00b5580 2024-04-25T14:04:18.045 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.045+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f95f006e500 con 0x7f95f00b5580 2024-04-25T14:04:18.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.697+0000 7f95e5ffb640 1 -- 172.21.15.150:0/579166929 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) ==== 104+0+0 (secure 0 0 0) 0x7f95ec062ad0 con 0x7f95f00b5580 2024-04-25T14:04:18.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 msgr2=0x7f95d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:18.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 0x7f95d40769c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f95e0001eb0 tx=0x7f95e0009040 comp rx=0 tx=0).stop 2024-04-25T14:04:18.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 msgr2=0x7f95f00cc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f95ec00c3c0 tx=0x7f95ec00c880 comp rx=0 tx=0).stop 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 shutdown_connections 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f95d40744b0 0x7f95d40769c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f95f0163e60 0x7f95f015bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f95f0161410 0x7f95f00b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 --2- 172.21.15.150:0/579166929 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f95f00b5580 0x7f95f00cc690 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 >> 172.21.15.150:0/579166929 conn(0x7f95f00d0580 msgr2=0x7f95f00b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 shutdown_connections 2024-04-25T14:04:18.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.700+0000 7f95f76d8640 1 -- 172.21.15.150:0/579166929 wait complete. 2024-04-25T14:04:18.713 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T14:04:18.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.827+0000 7fe80db00640 1 Processor -- start 2024-04-25T14:04:18.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.827+0000 7fe80db00640 1 -- start start 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe80db00640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe808160470 0x7fe8080b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe80db00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080c9460 0x7fe8080d0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe80db00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8080d0570 con 0x7fe8080b41e0 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe80db00640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe8080b4990 con 0x7fe808160470 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe80db00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8080b4c70 con 0x7fe8080c9460 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe808160470 0x7fe8080b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe808160470 0x7fe8080b3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34964/0 (socket says 172.21.15.150:34964) 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 -- 172.21.15.150:0/2149256377 learned_addr learned my addr 172.21.15.150:0/2149256377 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 -- 172.21.15.150:0/2149256377 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080c9460 msgr2=0x7fe8080d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 --2- 172.21.15.150:0/2149256377 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080c9460 0x7fe8080d0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 -- 172.21.15.150:0/2149256377 >> v1:172.21.15.187:6789/0 conn(0x7fe8080b41e0 legacy=0x7fe8080c8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:18.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.829+0000 7fe8077fe640 1 -- 172.21.15.150:0/2149256377 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8080d0810 con 0x7fe808160470 2024-04-25T14:04:18.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe8077fe640 1 --2- 172.21.15.150:0/2149256377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8080b3ca0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009870 tx=0x7fe7f802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c14a5ab4ef25eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:18.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe8067fc640 1 -- 172.21.15.150:0/2149256377 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8037070 con 0x7fe808160470 2024-04-25T14:04:18.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe8067fc640 1 -- 172.21.15.150:0/2149256377 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7f8032440 con 0x7fe808160470 2024-04-25T14:04:18.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe8067fc640 1 -- 172.21.15.150:0/2149256377 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7f8032810 con 0x7fe808160470 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe80db00640 1 -- 172.21.15.150:0/2149256377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 msgr2=0x7fe8080b3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.830+0000 7fe80db00640 1 --2- 172.21.15.150:0/2149256377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8080b3ca0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8009870 tx=0x7fe7f802a2c0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 -- 172.21.15.150:0/2149256377 shutdown_connections 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 --2- 172.21.15.150:0/2149256377 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080c9460 0x7fe8080d0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 --2- 172.21.15.150:0/2149256377 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8080b3ca0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 -- 172.21.15.150:0/2149256377 >> 172.21.15.150:0/2149256377 conn(0x7fe8080b76d0 msgr2=0x7fe8080b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 -- 172.21.15.150:0/2149256377 shutdown_connections 2024-04-25T14:04:18.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.831+0000 7fe80db00640 1 -- 172.21.15.150:0/2149256377 wait complete. 2024-04-25T14:04:18.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.832+0000 7fe80db00640 1 Processor -- start 2024-04-25T14:04:18.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.833+0000 7fe80db00640 1 -- start start 2024-04-25T14:04:18.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080b41e0 0x7fe8081f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080c9460 0x7fe80806e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe8080b4770 con 0x7fe808160470 2024-04-25T14:04:18.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe8080b45f0 con 0x7fe8080c9460 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe80db00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe8080ccf50 con 0x7fe8080b41e0 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34968/0 (socket says 172.21.15.150:34968) 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 -- 172.21.15.150:0/3627492594 learned_addr learned my addr 172.21.15.150:0/3627492594 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe8077fe640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080b41e0 0x7fe8081f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe806ffd640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080c9460 0x7fe80806e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 -- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080b41e0 msgr2=0x7fe8081f6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080b41e0 0x7fe8081f6400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 -- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080c9460 msgr2=0x7fe80806e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080c9460 0x7fe80806e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 -- 172.21.15.150:0/3627492594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe80806d690 con 0x7fe808160470 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.834+0000 7fe807fff640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe80000ba10 tx=0x7fe80000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.835+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800019070 con 0x7fe808160470 2024-04-25T14:04:18.836 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.835+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe80000cb40 con 0x7fe808160470 2024-04-25T14:04:18.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.835+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe80806d970 con 0x7fe808160470 2024-04-25T14:04:18.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.835+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe800014440 con 0x7fe808160470 2024-04-25T14:04:18.837 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.835+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe80806df30 con 0x7fe808160470 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.836+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe808050740 con 0x7fe808160470 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.836+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe80001c070 con 0x7fe808160470 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.837+0000 7fe7effff640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 0x7fe7f40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.837+0000 7fe8077fe640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 0x7fe7f40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.837+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 6388+0+0 (secure 0 0 0) 0x7fe800095100 con 0x7fe808160470 2024-04-25T14:04:18.838 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.837+0000 7fe8077fe640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 0x7fe7f40769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8034000 tx=0x7fe7f802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:18.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:18.842+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe80005ef90 con 0x7fe808160470 2024-04-25T14:04:19.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.017+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe8080c90d0 con 0x7fe808160470 2024-04-25T14:04:19.817 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.817+0000 7fe7effff640 1 -- 172.21.15.150:0/3627492594 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v46) ==== 181+0+0 (secure 0 0 0) 0x7fe800063290 con 0x7fe808160470 2024-04-25T14:04:19.818 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-04-25T14:04:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 msgr2=0x7fe7f40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 0x7fe7f40769c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fe7f8034000 tx=0x7fe7f802e000 comp rx=0 tx=0).stop 2024-04-25T14:04:19.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 msgr2=0x7fe8081632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe80000ba10 tx=0x7fe80000bed0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 shutdown_connections 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe7f40744b0 0x7fe7f40769c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe808160470 0x7fe8081632e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe8080c9460 0x7fe80806e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 --2- 172.21.15.150:0/3627492594 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe8080b41e0 0x7fe8081f6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 >> 172.21.15.150:0/3627492594 conn(0x7fe8080b76d0 msgr2=0x7fe8080ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.821+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 shutdown_connections 2024-04-25T14:04:19.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.822+0000 7fe80db00640 1 -- 172.21.15.150:0/3627492594 wait complete. 2024-04-25T14:04:19.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T14:04:19.953 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.953+0000 7f193c92d640 1 Processor -- start 2024-04-25T14:04:19.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.953+0000 7f193c92d640 1 -- start start 2024-04-25T14:04:19.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193c92d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f19380d0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193c92d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f193815e0f0 0x7f19380b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193c92d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19380d1750 con 0x7f193815d5d0 2024-04-25T14:04:19.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193c92d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f19380af3d0 con 0x7f193815e0f0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193c92d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19380d14f0 con 0x7f1938154230 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f1935d9b640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f193815d5d0 0x7f193815d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45558/0 (socket says 172.21.15.150:45558) 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f1935d9b640 1 -- 172.21.15.150:0/1691881383 learned_addr learned my addr 172.21.15.150:0/1691881383 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364981646 0 0) 0x7f19380d1750 con 0x7f193815d5d0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1920003520 con 0x7f193815d5d0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f193659c640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f19380d0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.955+0000 7f1936d9d640 1 --2- 172.21.15.150:0/1691881383 >> v2:172.21.15.187:3300/0 conn(0x7f193815e0f0 0x7f19380b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1961316606 0 0) 0x7f1920003520 con 0x7f193815d5d0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 msgr2=0x7f19380d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f19380d0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 >> v2:172.21.15.187:3300/0 conn(0x7f193815e0f0 msgr2=0x7f19380b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 --2- 172.21.15.150:0/1691881383 >> v2:172.21.15.187:3300/0 conn(0x7f193815e0f0 0x7f19380b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19380aeaa0 con 0x7f193815d5d0 2024-04-25T14:04:19.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f192c001670 con 0x7f193815d5d0 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 >> v1:172.21.15.150:6789/0 conn(0x7f193815d5d0 legacy=0x7f193815d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19200039b0 0x7f1920003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f1920004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1920004d40 0x7f1920005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19380af3d0 con 0x7f19200039b0 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19380d14f0 con 0x7f1920004d40 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f192000ba40 con 0x7f1938154230 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19200039b0 0x7f1920003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f1935d9b640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f1920004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f1936d9d640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1920004d40 0x7f1920005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 -- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 msgr2=0x7f1920004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1938154230 0x7f1920004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.959 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 -- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1920004d40 msgr2=0x7f1920005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1920004d40 0x7f1920005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f193659c640 1 -- 172.21.15.150:0/1691881383 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1920003520 con 0x7f19200039b0 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.956+0000 7f1936d9d640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1920004d40 0x7f1920005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.957+0000 7f193659c640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19200039b0 0x7f1920003e20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fc0 tx=0x7f1920005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416867b1d53f177f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.957+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928034020 con 0x7f19200039b0 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.957+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19280305f0 con 0x7f19200039b0 2024-04-25T14:04:19.960 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.957+0000 7f193559a640 1 -- 172.21.15.150:0/1691881383 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1928009c10 con 0x7f19200039b0 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 -- 172.21.15.150:0/1691881383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19200039b0 msgr2=0x7f1920003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 --2- 172.21.15.150:0/1691881383 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19200039b0 0x7f1920003e20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fc0 tx=0x7f1920005830 comp rx=0 tx=0).stop 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f1936d9d640 1 -- 172.21.15.150:0/1691881383 reap_dead start 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 -- 172.21.15.150:0/1691881383 shutdown_connections 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 -- 172.21.15.150:0/1691881383 >> 172.21.15.150:0/1691881383 conn(0x7f19380743b0 msgr2=0x7f193806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 -- 172.21.15.150:0/1691881383 shutdown_connections 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.958+0000 7f193c92d640 1 -- 172.21.15.150:0/1691881383 wait complete. 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.960+0000 7f193c92d640 1 Processor -- start 2024-04-25T14:04:19.963 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.960+0000 7f193c92d640 1 -- start start 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 0x7f19381461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 0x7f19380b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19380af860 con 0x7f193815e990 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19380af060 con 0x7f19200039b0 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193c92d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19380d0fb0 con 0x7f193815f2e0 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34982/0 (socket says 172.21.15.150:34982) 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 -- 172.21.15.150:0/262675549 learned_addr learned my addr 172.21.15.150:0/262675549 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1936d9d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 0x7f19380b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193659c640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 0x7f19381461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 -- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 msgr2=0x7f19380b5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 0x7f19380b5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.964 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 -- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 msgr2=0x7f19381461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 0x7f19381461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1935d9b640 1 -- 172.21.15.150:0/262675549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19380b64c0 con 0x7f193815e990 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f1936d9d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 0x7f19380b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.961+0000 7f193659c640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 0x7f19381461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f1935d9b640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f192c007290 tx=0x7f192c007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00f9c0 con 0x7f193815e990 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f192c018070 con 0x7f193815e990 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19380b67a0 con 0x7f193815e990 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192c00fda0 con 0x7f193815e990 2024-04-25T14:04:19.965 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.962+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1938146720 con 0x7f193815e990 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.963+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1938160690 con 0x7f193815e990 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.964+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f192c01d460 con 0x7f193815e990 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.964+0000 7f1926ffd640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 0x7f19180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.964+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5950+0+0 (secure 0 0 0) 0x7f192c095380 con 0x7f193815e990 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.964+0000 7f193659c640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 0x7f19180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:19.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.964+0000 7f193659c640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 0x7f19180769c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1928035000 tx=0x7f192802f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:19.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:19.969+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1938160690 con 0x7f193815e990 2024-04-25T14:04:20.146 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.146+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f193815cfc0 con 0x7f193815e990 2024-04-25T14:04:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.823+0000 7f1926ffd640 1 -- 172.21.15.150:0/262675549 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v47) ==== 169+0+0 (secure 0 0 0) 0x7f193815cfc0 con 0x7f193815e990 2024-04-25T14:04:20.823 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.828+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 msgr2=0x7f19180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.828+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 0x7f19180769c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1928035000 tx=0x7f192802f000 comp rx=0 tx=0).stop 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 msgr2=0x7f193815eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f192c007290 tx=0x7f192c007750 comp rx=0 tx=0).stop 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 shutdown_connections 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f19180744b0 0x7f19180769c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f193815f2e0 0x7f19380b5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:20.828 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f193815e990 0x7f193815eda0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:20.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 --2- 172.21.15.150:0/262675549 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19200039b0 0x7f19381461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:20.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 >> 172.21.15.150:0/262675549 conn(0x7f19380743b0 msgr2=0x7f193815c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:20.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 shutdown_connections 2024-04-25T14:04:20.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:20.829+0000 7f193c92d640 1 -- 172.21.15.150:0/262675549 wait complete. 2024-04-25T14:04:20.852 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:04:20.854 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:04:20.895 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-25T14:04:20.895 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-25T14:04:20.895 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:04:20.895 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:04:20.896 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.897 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-25T14:04:20.898 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-25T14:04:20.899 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-25T14:04:20.900 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-25T14:04:20.901 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-25T14:04:20.902 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.903 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:04:20.904 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-04-25T14:04:20.905 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-25T14:04:20.906 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-25T14:04:20.907 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-25T14:04:20.908 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-25T14:04:20.909 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-25T14:04:20.910 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-25T14:04:20.911 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-25T14:04:20.912 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:20.913 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-25T14:04:20.914 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:20.915 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-25T14:04:20.917 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:04:20.917 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:04:20.917 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-25T14:04:20.917 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-25T14:04:20.917 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:04:20.918 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:04:20.919 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:04:20.920 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T14:04:21.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.071+0000 7f3c7430e640 1 Processor -- start 2024-04-25T14:04:21.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.071+0000 7f3c7430e640 1 -- start start 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c7430e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0d3290 0x7f3c6c160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c7430e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c6c0735b0 0x7f3c6c0ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c7430e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c6c164e10 con 0x7f3c6c0cf330 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c7430e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f3c6c160770 con 0x7f3c6c0735b0 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c7430e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6c0b6c70 con 0x7f3c6c0d3290 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c6c0735b0 0x7f3c6c0ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f3c6c0735b0 0x7f3c6c0ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:34988/0 (socket says 172.21.15.150:34988) 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 -- 172.21.15.150:0/1224712919 learned_addr learned my addr 172.21.15.150:0/1224712919 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c720aa640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0d3290 0x7f3c6c160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 -- 172.21.15.150:0/1224712919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0d3290 msgr2=0x7f3c6c160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0d3290 0x7f3c6c160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 -- 172.21.15.150:0/1224712919 >> v1:172.21.15.187:6789/0 conn(0x7f3c6c0cf330 legacy=0x7f3c6c072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.073+0000 7f3c728ab640 1 -- 172.21.15.150:0/1224712919 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c164050 con 0x7f3c6c0735b0 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.074+0000 7f3c728ab640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0735b0 0x7f3c6c0ce000 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009870 tx=0x7f3c6402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd1299412bd0a729 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.074+0000 7f3c710a8640 1 -- 172.21.15.150:0/1224712919 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64037070 con 0x7f3c6c0735b0 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.074+0000 7f3c710a8640 1 -- 172.21.15.150:0/1224712919 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c64032440 con 0x7f3c6c0735b0 2024-04-25T14:04:21.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.074+0000 7f3c710a8640 1 -- 172.21.15.150:0/1224712919 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c64032810 con 0x7f3c6c0735b0 2024-04-25T14:04:21.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 -- 172.21.15.150:0/1224712919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0735b0 msgr2=0x7f3c6c0ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0735b0 0x7f3c6c0ce000 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009870 tx=0x7f3c6402a1c0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 -- 172.21.15.150:0/1224712919 shutdown_connections 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0735b0 0x7f3c6c0ce000 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 --2- 172.21.15.150:0/1224712919 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0d3290 0x7f3c6c160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 -- 172.21.15.150:0/1224712919 >> 172.21.15.150:0/1224712919 conn(0x7f3c6c0d39b0 msgr2=0x7f3c6c0d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 -- 172.21.15.150:0/1224712919 shutdown_connections 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.075+0000 7f3c7430e640 1 -- 172.21.15.150:0/1224712919 wait complete. 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.076+0000 7f3c7430e640 1 Processor -- start 2024-04-25T14:04:21.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.077+0000 7f3c7430e640 1 -- start start 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c6c0b0730 0x7f3c6c0b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0b3ae0 0x7f3c6c0b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c1648f0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6c164610 con 0x7f3c6c0b0730 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c7430e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f3c6c0b6800 con 0x7f3c6c0b3ae0 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35000/0 (socket says 172.21.15.150:35000) 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 -- 172.21.15.150:0/384270992 learned_addr learned my addr 172.21.15.150:0/384270992 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 -- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0b3ae0 msgr2=0x7f3c6c0b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c728ab640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0b3ae0 0x7f3c6c0b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c718a9640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c6c0b0730 0x7f3c6c0b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0b3ae0 0x7f3c6c0b3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 -- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c6c0b0730 msgr2=0x7f3c6c0b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c6c0b0730 0x7f3c6c0b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 -- 172.21.15.150:0/384270992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c6c0b13b0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.078+0000 7f3c720aa640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3c68007e90 tx=0x7f3c6800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.079+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68014620 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.079+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c68019070 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.079+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68014a20 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.079+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c6c0b1690 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.079+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c6c0b1c50 con 0x7f3c6c0cf330 2024-04-25T14:04:21.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.080+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c6c076a40 con 0x7f3c6c0cf330 2024-04-25T14:04:21.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.080+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f3c68014bc0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.081+0000 7f3c62ffd640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 0x7f3c500769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.081+0000 7f3c718a9640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 0x7f3c500769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.081+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f3c68095aa0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.081+0000 7f3c718a9640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 0x7f3c500769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0b3870 tx=0x7f3c5c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.086+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3c680602f0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.261+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f3c6c15f8d0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.265+0000 7f3c62ffd640 1 -- 172.21.15.150:0/384270992 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f3c6805fbf0 con 0x7f3c6c0cf330 2024-04-25T14:04:21.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 msgr2=0x7f3c500769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 0x7f3c500769c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0b3870 tx=0x7f3c5c009040 comp rx=0 tx=0).stop 2024-04-25T14:04:21.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 msgr2=0x7f3c6c14cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f3c68007e90 tx=0x7f3c6800e4b0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 shutdown_connections 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f3c500744b0 0x7f3c500769c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f3c6c0b3ae0 0x7f3c6c0b3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f3c6c0b0730 0x7f3c6c0b0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 --2- 172.21.15.150:0/384270992 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3c6c0cf330 0x7f3c6c14cf70 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 >> 172.21.15.150:0/384270992 conn(0x7f3c6c0d39b0 msgr2=0x7f3c6c0d1940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 shutdown_connections 2024-04-25T14:04:21.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.267+0000 7f3c7430e640 1 -- 172.21.15.150:0/384270992 wait complete. 2024-04-25T14:04:21.277 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj' 2024-04-25T14:04:21.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.386+0000 7f69d669f640 1 Processor -- start 2024-04-25T14:04:21.385 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.386+0000 7f69d669f640 1 -- start start 2024-04-25T14:04:21.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69d669f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f69d00b2650 0x7f69d0160160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69d669f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d00b2de0 con 0x7f69d00b74a0 2024-04-25T14:04:21.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69d669f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69d00b4550 con 0x7f69d00b78b0 2024-04-25T14:04:21.387 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69d669f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f69d00b39b0 con 0x7f69d00b2650 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f69d00b2650 0x7f69d0160160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cf7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f69d00b74a0 0x7f69d00b3e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45568/0 (socket says 172.21.15.150:45568) 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f69d00b2650 0x7f69d0160160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35016/0 (socket says 172.21.15.150:35016) 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cf7fe640 1 -- 172.21.15.150:0/1113111945 learned_addr learned my addr 172.21.15.150:0/1113111945 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cffff640 1 -- 172.21.15.150:0/1113111945 >> v1:172.21.15.187:6789/0 conn(0x7f69d00b78b0 legacy=0x7f69d006cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cffff640 1 -- 172.21.15.150:0/1113111945 >> v1:172.21.15.150:6789/0 conn(0x7f69d00b74a0 legacy=0x7f69d00b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69ceffd640 1 -- 172.21.15.150:0/1113111945 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2480263400 0 0) 0x7f69d00b2de0 con 0x7f69d00b74a0 2024-04-25T14:04:21.388 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.388+0000 7f69cffff640 1 -- 172.21.15.150:0/1113111945 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d00b3080 con 0x7f69d00b2650 2024-04-25T14:04:21.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.389+0000 7f69cffff640 1 --2- 172.21.15.150:0/1113111945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b2650 0x7f69d0160160 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f69c00097a0 tx=0x7f69c002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46270cae42c802da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.389+0000 7f69ceffd640 1 -- 172.21.15.150:0/1113111945 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0037070 con 0x7f69d00b2650 2024-04-25T14:04:21.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.389+0000 7f69ceffd640 1 -- 172.21.15.150:0/1113111945 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c0032440 con 0x7f69d00b2650 2024-04-25T14:04:21.389 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.389+0000 7f69ceffd640 1 -- 172.21.15.150:0/1113111945 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0032810 con 0x7f69d00b2650 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 -- 172.21.15.150:0/1113111945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b2650 msgr2=0x7f69d0160160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 --2- 172.21.15.150:0/1113111945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b2650 0x7f69d0160160 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f69c00097a0 tx=0x7f69c002a1a0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 -- 172.21.15.150:0/1113111945 shutdown_connections 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 --2- 172.21.15.150:0/1113111945 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b2650 0x7f69d0160160 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 -- 172.21.15.150:0/1113111945 >> 172.21.15.150:0/1113111945 conn(0x7f69d0073ce0 msgr2=0x7f69d009b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 -- 172.21.15.150:0/1113111945 shutdown_connections 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.390+0000 7f69d669f640 1 -- 172.21.15.150:0/1113111945 wait complete. 2024-04-25T14:04:21.390 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.391+0000 7f69d669f640 1 Processor -- start 2024-04-25T14:04:21.391 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.392+0000 7f69d669f640 1 -- start start 2024-04-25T14:04:21.393 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69d00b2650 0x7f69d015bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69d00b78b0 0x7f69d0149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f69d006d300 con 0x7f69d00b74a0 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f69d00b3640 con 0x7f69d00b78b0 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d669f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f69d006d480 con 0x7f69d00b2650 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35028/0 (socket says 172.21.15.150:35028) 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 -- 172.21.15.150:0/2807261865 learned_addr learned my addr 172.21.15.150:0/2807261865 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cffff640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69d00b2650 0x7f69d015bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.394 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69d4c3c640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69d00b78b0 0x7f69d0149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 -- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69d00b2650 msgr2=0x7f69d015bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69d00b2650 0x7f69d015bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 -- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69d00b78b0 msgr2=0x7f69d0149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69d00b78b0 0x7f69d0149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.393+0000 7f69cf7fe640 1 -- 172.21.15.150:0/2807261865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69d015c500 con 0x7f69d00b74a0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69cf7fe640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c420 tx=0x7f69c400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c40074e0 con 0x7f69d00b74a0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c40047e0 con 0x7f69d00b74a0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69d00cd0d0 con 0x7f69d00b74a0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c4004bd0 con 0x7f69d00b74a0 2024-04-25T14:04:21.395 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.394+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69d00cd690 con 0x7f69d00b74a0 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.395+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69d0076a40 con 0x7f69d00b74a0 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.396+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f69c4011070 con 0x7f69d00b74a0 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.396+0000 7f69ccff9640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 0x7f69bc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.396+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f69c4094c10 con 0x7f69d00b74a0 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.396+0000 7f69cffff640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 0x7f69bc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.397+0000 7f69cffff640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 0x7f69bc0769f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f69c0034000 tx=0x7f69c002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.401+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f69c405eea0 con 0x7f69d00b74a0 2024-04-25T14:04:21.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.643+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj"]} v 0) -- 0x7f69d00b4750 con 0x7f69d00b74a0 2024-04-25T14:04:21.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.824+0000 7f69ccff9640 1 -- 172.21.15.150:0/2807261865 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_max_items_per_obj"]}]=0 v136) ==== 147+0+0 (secure 0 0 0) 0x7f69c40631a0 con 0x7f69d00b74a0 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.826+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 msgr2=0x7f69bc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.826+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 0x7f69bc0769f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f69c0034000 tx=0x7f69c002e000 comp rx=0 tx=0).stop 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.826+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 msgr2=0x7f69d0162fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.826+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f69c400c420 tx=0x7f69c400c8e0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 shutdown_connections 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f69bc0744e0 0x7f69bc0769f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f69d00b78b0 0x7f69d0149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f69d00b74a0 0x7f69d0162fb0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 --2- 172.21.15.150:0/2807261865 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f69d00b2650 0x7f69d015bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 >> 172.21.15.150:0/2807261865 conn(0x7f69d0073ce0 msgr2=0x7f69d006ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:21.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 shutdown_connections 2024-04-25T14:04:21.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.827+0000 7f69d669f640 1 -- 172.21.15.150:0/2807261865 wait complete. 2024-04-25T14:04:21.838 INFO:tasks.cephfs_test_runner:The maximum number of openfiles omap objects keys are now equal to ... ok 2024-04-25T14:04:21.838 INFO:tasks.cephfs_test_runner:Starting test: test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable) Opening a file should increment omap_total_updates by 1. 2024-04-25T14:04:21.838 INFO:tasks.cephfs_test_runner:test_perf_counters (tasks.cephfs.test_openfiletable.OpenFileTable) 2024-04-25T14:04:21.839 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:21.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.948+0000 7f94eb20d640 1 Processor -- start 2024-04-25T14:04:21.947 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.948+0000 7f94eb20d640 1 -- start start 2024-04-25T14:04:21.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94eb20d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94e40cfd10 0x7f94e40b4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.948 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94eb20d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e40b3c30 con 0x7f94e40d22e0 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94eb20d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94e40b3f10 con 0x7f94e406f500 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94eb20d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f94e40b39e0 con 0x7f94e40cfd10 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e8fa9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94e40cfd10 0x7f94e40b4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e97aa640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f94e40d22e0 0x7f94e40b1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45580/0 (socket says 172.21.15.150:45580) 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e97aa640 1 -- 172.21.15.150:0/1439009432 learned_addr learned my addr 172.21.15.150:0/1439009432 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e8fa9640 1 -- 172.21.15.150:0/1439009432 >> v1:172.21.15.187:6789/0 conn(0x7f94e406f500 legacy=0x7f94e40d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e8fa9640 1 -- 172.21.15.150:0/1439009432 >> v1:172.21.15.150:6789/0 conn(0x7f94e40d22e0 legacy=0x7f94e40b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e37fe640 1 -- 172.21.15.150:0/1439009432 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3716474431 0 0) 0x7f94e40b3c30 con 0x7f94e40d22e0 2024-04-25T14:04:21.949 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.949+0000 7f94e8fa9640 1 -- 172.21.15.150:0/1439009432 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e40b41f0 con 0x7f94e40cfd10 2024-04-25T14:04:21.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.950+0000 7f94e8fa9640 1 --2- 172.21.15.150:0/1439009432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40cfd10 0x7f94e40b4cd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f94d80097a0 tx=0x7f94d802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c2164e2f85fecf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.950+0000 7f94e37fe640 1 -- 172.21.15.150:0/1439009432 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8037070 con 0x7f94e40cfd10 2024-04-25T14:04:21.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.950+0000 7f94e37fe640 1 -- 172.21.15.150:0/1439009432 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d8032440 con 0x7f94e40cfd10 2024-04-25T14:04:21.950 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.950+0000 7f94e37fe640 1 -- 172.21.15.150:0/1439009432 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8032810 con 0x7f94e40cfd10 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 -- 172.21.15.150:0/1439009432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40cfd10 msgr2=0x7f94e40b4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 --2- 172.21.15.150:0/1439009432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40cfd10 0x7f94e40b4cd0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f94d80097a0 tx=0x7f94d802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 -- 172.21.15.150:0/1439009432 shutdown_connections 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 --2- 172.21.15.150:0/1439009432 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e40cfd10 0x7f94e40b4cd0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 -- 172.21.15.150:0/1439009432 >> 172.21.15.150:0/1439009432 conn(0x7f94e409b4a0 msgr2=0x7f94e409b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 -- 172.21.15.150:0/1439009432 shutdown_connections 2024-04-25T14:04:21.951 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.951+0000 7f94eb20d640 1 -- 172.21.15.150:0/1439009432 wait complete. 2024-04-25T14:04:21.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.953+0000 7f94eb20d640 1 Processor -- start 2024-04-25T14:04:21.952 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.953+0000 7f94eb20d640 1 -- start start 2024-04-25T14:04:21.954 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94e40cfd10 0x7f94e40c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94e40d22e0 0x7f94e4162180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94e406f910 con 0x7f94e406f500 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f94e40b47b0 con 0x7f94e40cfd10 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94eb20d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f94e4164a80 con 0x7f94e40d22e0 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94e8fa9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.955 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94e8fa9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35048/0 (socket says 172.21.15.150:35048) 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94e8fa9640 1 -- 172.21.15.150:0/2140699995 learned_addr learned my addr 172.21.15.150:0/2140699995 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94e97aa640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94e40d22e0 0x7f94e4162180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.954+0000 7f94e3fff640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94e40cfd10 0x7f94e40c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 -- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94e40d22e0 msgr2=0x7f94e4162180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94e40d22e0 0x7f94e4162180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 -- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94e40cfd10 msgr2=0x7f94e40c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94e40cfd10 0x7f94e40c6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 -- 172.21.15.150:0/2140699995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94e41f9c00 con 0x7f94e406f500 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e8fa9640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f94d80097a0 tx=0x7f94d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8041070 con 0x7f94e406f500 2024-04-25T14:04:21.956 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94d803c870 con 0x7f94e406f500 2024-04-25T14:04:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94e41f9ee0 con 0x7f94e406f500 2024-04-25T14:04:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94d8037040 con 0x7f94e406f500 2024-04-25T14:04:21.957 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.955+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94e41fa4a0 con 0x7f94e406f500 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.956+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94e4076a40 con 0x7f94e406f500 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.957+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f94d8044070 con 0x7f94e406f500 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.957+0000 7f94e17fa640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 0x7f94c80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.957+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f94d80b5cf0 con 0x7f94e406f500 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.957+0000 7f94e3fff640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 0x7f94c80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:21.958 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.958+0000 7f94e3fff640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 0x7f94c80769f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f94e40b2960 tx=0x7f94d0009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:21.961 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:21.962+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94d80ba070 con 0x7f94e406f500 2024-04-25T14:04:22.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.136+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94e40cf3b0 con 0x7f94e406f500 2024-04-25T14:04:22.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.137+0000 7f94e17fa640 1 -- 172.21.15.150:0/2140699995 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+13559 (secure 0 0 0) 0x7f94d8080540 con 0x7f94e406f500 2024-04-25T14:04:22.136 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:22.137 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":47,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:20.819218+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6835/4048124689":"2024-04-26T14:04:17.696090+0000","172.21.15.187:6833/1241446695":"2024-04-26T14:03:35.677482+0000","172.21.15.187:6833/4048124689":"2024-04-26T14:04:17.696090+0000","172.21.15.150:6836/3196344346":"2024-04-26T14:03:34.944594+0000","172.21.15.150:6834/3196344346":"2024-04-26T14:03:34.944594+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 msgr2=0x7f94c80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 0x7f94c80769f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f94e40b2960 tx=0x7f94d0009040 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 msgr2=0x7f94e41538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f94d80097a0 tx=0x7f94d80046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 shutdown_connections 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94c80744e0 0x7f94c80769f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94e40d22e0 0x7f94e4162180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94e40cfd10 0x7f94e40c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 --2- 172.21.15.150:0/2140699995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94e406f500 0x7f94e41538a0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 >> 172.21.15.150:0/2140699995 conn(0x7f94e409b4a0 msgr2=0x7f94e40b1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 shutdown_connections 2024-04-25T14:04:22.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.139+0000 7f94eb20d640 1 -- 172.21.15.150:0/2140699995 wait complete. 2024-04-25T14:04:22.150 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:04:22.150 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:04:22.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.262+0000 7fc145e15640 1 Processor -- start 2024-04-25T14:04:22.262 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.263+0000 7fc145e15640 1 -- start start 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1401603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc140072ea0 0x7fc1400732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc1400d18b0 0x7fc1400d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc1400d2e90 con 0x7fc140072ea0 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc1400d0080 con 0x7fc1400d18b0 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc145e15640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400737f0 con 0x7fc1400b6800 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc140072ea0 0x7fc1400732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc140072ea0 0x7fc1400732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35056/0 (socket says 172.21.15.150:35056) 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 -- 172.21.15.150:0/210041105 learned_addr learned my addr 172.21.15.150:0/210041105 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13f7fe640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1401603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13ffff640 1 --2- 172.21.15.150:0/210041105 >> v2:172.21.15.187:3300/0 conn(0x7fc1400d18b0 0x7fc1400d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 -- 172.21.15.150:0/210041105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 msgr2=0x7fc1401603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:22.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1401603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 -- 172.21.15.150:0/210041105 >> v2:172.21.15.187:3300/0 conn(0x7fc1400d18b0 msgr2=0x7fc1400d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 --2- 172.21.15.150:0/210041105 >> v2:172.21.15.187:3300/0 conn(0x7fc1400d18b0 0x7fc1400d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13effd640 1 -- 172.21.15.150:0/210041105 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc140164a50 con 0x7fc140072ea0 2024-04-25T14:04:22.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.264+0000 7fc13f7fe640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1401603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.265+0000 7fc13effd640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1400732b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc1400739a0 tx=0x7fc13402a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e52467b05d98c60d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.265+0000 7fc13e7fc640 1 -- 172.21.15.150:0/210041105 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134037070 con 0x7fc140072ea0 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.265+0000 7fc13e7fc640 1 -- 172.21.15.150:0/210041105 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc134032440 con 0x7fc140072ea0 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.265+0000 7fc13e7fc640 1 -- 172.21.15.150:0/210041105 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134032810 con 0x7fc140072ea0 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 -- 172.21.15.150:0/210041105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 msgr2=0x7fc1400732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1400732b0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fc1400739a0 tx=0x7fc13402a130 comp rx=0 tx=0).stop 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 -- 172.21.15.150:0/210041105 shutdown_connections 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 --2- 172.21.15.150:0/210041105 >> v2:172.21.15.187:3300/0 conn(0x7fc1400d18b0 0x7fc1400d1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1400732b0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 --2- 172.21.15.150:0/210041105 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1401603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 -- 172.21.15.150:0/210041105 >> 172.21.15.150:0/210041105 conn(0x7fc1400d39b0 msgr2=0x7fc1400d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 -- 172.21.15.150:0/210041105 shutdown_connections 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.266+0000 7fc145e15640 1 -- 172.21.15.150:0/210041105 wait complete. 2024-04-25T14:04:22.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.268+0000 7fc145e15640 1 Processor -- start 2024-04-25T14:04:22.268 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.268+0000 7fc145e15640 1 -- start start 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1400b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1400d18b0 0x7fc1401629b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400cfd10 con 0x7fc140072ea0 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc1400cfa30 con 0x7fc1400d18b0 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc145e15640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc1400b98e0 con 0x7fc1400b6800 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc13f7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35070/0 (socket says 172.21.15.150:35070) 2024-04-25T14:04:22.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc13f7fe640 1 -- 172.21.15.150:0/2184587675 learned_addr learned my addr 172.21.15.150:0/2184587675 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc13effd640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1400b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.269+0000 7fc13ffff640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1400d18b0 0x7fc1401629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 -- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 msgr2=0x7fc1400b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1400b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 -- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1400d18b0 msgr2=0x7fc1401629b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1400d18b0 0x7fc1401629b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 -- 172.21.15.150:0/2184587675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14015a740 con 0x7fc140072ea0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc13f7fe640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c820 tx=0x7fc13000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130015070 con 0x7fc140072ea0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1300077d0 con 0x7fc140072ea0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1400b2570 con 0x7fc140072ea0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130007bc0 con 0x7fc140072ea0 2024-04-25T14:04:22.271 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.270+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1400b2b30 con 0x7fc140072ea0 2024-04-25T14:04:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.271+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1400b6de0 con 0x7fc140072ea0 2024-04-25T14:04:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.272+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc130018070 con 0x7fc140072ea0 2024-04-25T14:04:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.272+0000 7fc127fff640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 0x7fc12c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.272+0000 7fc13effd640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 0x7fc12c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.272+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7fc130094e90 con 0x7fc140072ea0 2024-04-25T14:04:22.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.273+0000 7fc13effd640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 0x7fc12c0769c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc134034000 tx=0x7fc13402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.277+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc1400b6de0 con 0x7fc140072ea0 2024-04-25T14:04:22.449 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.450+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc1400b22d0 con 0x7fc140072ea0 2024-04-25T14:04:22.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.450+0000 7fc127fff640 1 -- 172.21.15.150:0/2184587675 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) ==== 93+0+10 (secure 0 0 0) 0x7fc1400b22d0 con 0x7fc140072ea0 2024-04-25T14:04:22.450 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 msgr2=0x7fc12c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 0x7fc12c0769c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc134034000 tx=0x7fc13402e000 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 msgr2=0x7fc1401579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc13000c820 tx=0x7fc13000cce0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 shutdown_connections 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc12c0744b0 0x7fc12c0769c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc1400d18b0 0x7fc1401629b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc1400b6800 0x7fc1400b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 --2- 172.21.15.150:0/2184587675 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc140072ea0 0x7fc1401579b0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.452+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 >> 172.21.15.150:0/2184587675 conn(0x7fc1400d39b0 msgr2=0x7fc14006f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.453+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 shutdown_connections 2024-04-25T14:04:22.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.453+0000 7fc145e15640 1 -- 172.21.15.150:0/2184587675 wait complete. 2024-04-25T14:04:22.463 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters' 2024-04-25T14:04:22.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.572+0000 7fd282867640 1 Processor -- start 2024-04-25T14:04:22.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.572+0000 7fd282867640 1 -- start start 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd282867640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd27c15ec30 0x7fd27c1625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd282867640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27c15f100 con 0x7fd27c0b5b10 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd282867640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd27c15f3e0 con 0x7fd27c158760 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd282867640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd27c070fd0 con 0x7fd27c15ec30 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd27b7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd27c0b5b10 0x7fd27c15e520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45586/0 (socket says 172.21.15.150:45586) 2024-04-25T14:04:22.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.573+0000 7fd27b7fe640 1 -- 172.21.15.150:0/3057999916 learned_addr learned my addr 172.21.15.150:0/3057999916 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd280e04640 1 --2- 172.21.15.150:0/3057999916 >> v2:172.21.15.187:3300/0 conn(0x7fd27c15ec30 0x7fd27c1625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1542111451 0 0) 0x7fd27c15f100 con 0x7fd27c0b5b10 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd264003520 con 0x7fd27c0b5b10 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd280e04640 1 -- 172.21.15.150:0/3057999916 >> v1:172.21.15.187:6789/0 conn(0x7fd27c158760 legacy=0x7fd27c0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3355118021 0 0) 0x7fd264003520 con 0x7fd27c0b5b10 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd280e04640 1 -- 172.21.15.150:0/3057999916 >> v1:172.21.15.150:6789/0 conn(0x7fd27c0b5b10 legacy=0x7fd27c15e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.574+0000 7fd280e04640 1 -- 172.21.15.150:0/3057999916 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c161b10 con 0x7fd27c15ec30 2024-04-25T14:04:22.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.575+0000 7fd280e04640 1 --2- 172.21.15.150:0/3057999916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c1625f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd274009d90 tx=0x7fd27402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54426c0fd880c9a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.575+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd26c001670 con 0x7fd27c15ec30 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.575+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd274007540 con 0x7fd27c15ec30 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.575+0000 7fd27affd640 1 -- 172.21.15.150:0/3057999916 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2740078f0 con 0x7fd27c15ec30 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 -- 172.21.15.150:0/3057999916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 msgr2=0x7fd27c1625f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 --2- 172.21.15.150:0/3057999916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c1625f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd274009d90 tx=0x7fd27402a680 comp rx=0 tx=0).stop 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 -- 172.21.15.150:0/3057999916 shutdown_connections 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 --2- 172.21.15.150:0/3057999916 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c1625f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 -- 172.21.15.150:0/3057999916 >> 172.21.15.150:0/3057999916 conn(0x7fd27c0d0400 msgr2=0x7fd27c0d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 -- 172.21.15.150:0/3057999916 shutdown_connections 2024-04-25T14:04:22.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.576+0000 7fd282867640 1 -- 172.21.15.150:0/3057999916 wait complete. 2024-04-25T14:04:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.577+0000 7fd282867640 1 Processor -- start 2024-04-25T14:04:22.577 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.578+0000 7fd282867640 1 -- start start 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 0x7fd27c15d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c0703d0 con 0x7fd27c158760 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd27c1620d0 con 0x7fd27c15ec30 2024-04-25T14:04:22.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd282867640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd27c0ce370 con 0x7fd27c15e770 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35084/0 (socket says 172.21.15.150:35084) 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 -- 172.21.15.150:0/1148363829 learned_addr learned my addr 172.21.15.150:0/1148363829 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27b7fe640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd280e04640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 0x7fd27c15d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 -- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 msgr2=0x7fd27c15d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 0x7fd27c15d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 -- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 msgr2=0x7fd27c150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27bfff640 1 -- 172.21.15.150:0/1148363829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c159720 con 0x7fd27c158760 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd27b7fe640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:22.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.579+0000 7fd280e04640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 0x7fd27c15d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd27bfff640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c3c0 tx=0x7fd27000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270017070 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2700052d0 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd27c159a00 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd270003830 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.580+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd27c159fc0 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.581+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd27c076a40 con 0x7fd27c158760 2024-04-25T14:04:22.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.581+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd27001a070 con 0x7fd27c158760 2024-04-25T14:04:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.582+0000 7fd278ff9640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 0x7fd2680769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.582+0000 7fd27b7fe640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 0x7fd2680769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.582+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7fd270094480 con 0x7fd27c158760 2024-04-25T14:04:22.583 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.582+0000 7fd27b7fe640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 0x7fd2680769f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd27c15b9f0 tx=0x7fd26c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:22.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.586+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd27005ed60 con 0x7fd27c158760 2024-04-25T14:04:22.829 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:22.830+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters"]} v 0) -- 0x7fd27c0d1c80 con 0x7fd27c158760 2024-04-25T14:04:23.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.832+0000 7fd278ff9640 1 -- 172.21.15.150:0/1148363829 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters"]}]=0 v138) ==== 146+0+0 (secure 0 0 0) 0x7fd27005e660 con 0x7fd27c158760 2024-04-25T14:04:23.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 msgr2=0x7fd2680769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:23.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 0x7fd2680769f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd27c15b9f0 tx=0x7fd26c009040 comp rx=0 tx=0).stop 2024-04-25T14:04:23.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 msgr2=0x7fd27c06f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fd27000c3c0 tx=0x7fd27000c880 comp rx=0 tx=0).stop 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 shutdown_connections 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd2680744e0 0x7fd2680769f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd27c15e770 0x7fd27c15d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd27c15ec30 0x7fd27c150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 --2- 172.21.15.150:0/1148363829 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd27c158760 0x7fd27c06f8d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 >> 172.21.15.150:0/1148363829 conn(0x7fd27c0d0400 msgr2=0x7fd27c0b7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 shutdown_connections 2024-04-25T14:04:23.835 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.835+0000 7fd282867640 1 -- 172.21.15.150:0/1148363829 wait complete. 2024-04-25T14:04:23.849 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-25T14:04:23.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.967+0000 7f2346cd1640 1 Processor -- start 2024-04-25T14:04:23.966 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.967+0000 7f2346cd1640 1 -- start start 2024-04-25T14:04:23.968 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23400cf3d0 0x7f2340070120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 0x7f2340070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f2340162190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f23400d15d0 con 0x7f23400cf3d0 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f23400ce440 con 0x7f234015e360 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2346cd1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23400d1870 con 0x7f2340161d80 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2344a6d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23400cf3d0 0x7f2340070120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2344a6d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f23400cf3d0 0x7f2340070120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35090/0 (socket says 172.21.15.150:35090) 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f2344a6d640 1 -- 172.21.15.150:0/1004917163 learned_addr learned my addr 172.21.15.150:0/1004917163 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f234526e640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f2340162190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.968+0000 7f233ffff640 1 --2- 172.21.15.150:0/1004917163 >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 0x7f2340070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 -- 172.21.15.150:0/1004917163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 msgr2=0x7f2340162190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:23.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f2340162190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 -- 172.21.15.150:0/1004917163 >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 msgr2=0x7f2340070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 --2- 172.21.15.150:0/1004917163 >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 0x7f2340070660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 -- 172.21.15.150:0/1004917163 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23400d1b50 con 0x7f23400cf3d0 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f234526e640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f2340162190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f233ffff640 1 --2- 172.21.15.150:0/1004917163 >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 0x7f2340070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f2344a6d640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f2340070120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f23400754e0 tx=0x7f233002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9e389828b3c922 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f233f7fe640 1 -- 172.21.15.150:0/1004917163 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2330037070 con 0x7f23400cf3d0 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f233f7fe640 1 -- 172.21.15.150:0/1004917163 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2330032440 con 0x7f23400cf3d0 2024-04-25T14:04:23.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.969+0000 7f233f7fe640 1 -- 172.21.15.150:0/1004917163 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2330032810 con 0x7f23400cf3d0 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 -- 172.21.15.150:0/1004917163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 msgr2=0x7f2340070120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f2340070120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f23400754e0 tx=0x7f233002a1e0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 -- 172.21.15.150:0/1004917163 shutdown_connections 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f2340162190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 --2- 172.21.15.150:0/1004917163 >> v2:172.21.15.187:3300/0 conn(0x7f234015e360 0x7f2340070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 --2- 172.21.15.150:0/1004917163 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f2340070120 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 -- 172.21.15.150:0/1004917163 >> 172.21.15.150:0/1004917163 conn(0x7f23400d3960 msgr2=0x7f23400d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 -- 172.21.15.150:0/1004917163 shutdown_connections 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.970+0000 7f2346cd1640 1 -- 172.21.15.150:0/1004917163 wait complete. 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.972+0000 7f2346cd1640 1 Processor -- start 2024-04-25T14:04:23.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.972+0000 7f2346cd1640 1 -- start start 2024-04-25T14:04:23.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f234015e360 0x7f234015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f234015fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f2340070eb0 con 0x7f23400cf3d0 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f2340070d30 con 0x7f234015e360 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.973+0000 7f2346cd1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f23400cdfb0 con 0x7f2340161d80 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35092/0 (socket says 172.21.15.150:35092) 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 -- 172.21.15.150:0/4139244298 learned_addr learned my addr 172.21.15.150:0/4139244298 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f233ffff640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f234015e360 0x7f234015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f234526e640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f234015fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 -- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 msgr2=0x7f234015fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f234015fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 -- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f234015e360 msgr2=0x7f234015f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f234015e360 0x7f234015f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 -- 172.21.15.150:0/4139244298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f234015fb20 con 0x7f23400cf3d0 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f234526e640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f234015fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2344a6d640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f233002a220 tx=0x7f23300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2330041020 con 0x7f23400cf3d0 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23300306c0 con 0x7f23400cf3d0 2024-04-25T14:04:23.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2330037040 con 0x7f23400cf3d0 2024-04-25T14:04:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23400c7ad0 con 0x7f23400cf3d0 2024-04-25T14:04:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.974+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23400c8090 con 0x7f23400cf3d0 2024-04-25T14:04:23.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.975+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2340076a40 con 0x7f23400cf3d0 2024-04-25T14:04:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.976+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2330044020 con 0x7f23400cf3d0 2024-04-25T14:04:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.977+0000 7f233d7fa640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 0x7f232c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.977+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f23300b5ad0 con 0x7f23400cf3d0 2024-04-25T14:04:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.977+0000 7f233ffff640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 0x7f232c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:23.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.977+0000 7f233ffff640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 0x7f232c0769f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:23.983 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:23.981+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f233004b030 con 0x7f23400cf3d0 2024-04-25T14:04:24.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.159+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f234015a1b0 con 0x7f23400cf3d0 2024-04-25T14:04:24.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.163+0000 7f233d7fa640 1 -- 172.21.15.150:0/4139244298 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f2330080200 con 0x7f23400cf3d0 2024-04-25T14:04:24.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 msgr2=0x7f232c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 0x7f232c0769f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2334006fd0 tx=0x7f2334008040 comp rx=0 tx=0).stop 2024-04-25T14:04:24.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 msgr2=0x7f234014fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f233002a220 tx=0x7f23300046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 shutdown_connections 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f232c0744e0 0x7f232c0769f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2340161d80 0x7f234015fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f234015e360 0x7f234015f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 --2- 172.21.15.150:0/4139244298 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f23400cf3d0 0x7f234014fe50 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 >> 172.21.15.150:0/4139244298 conn(0x7f23400d3960 msgr2=0x7f23400684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 shutdown_connections 2024-04-25T14:04:24.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.165+0000 7f2346cd1640 1 -- 172.21.15.150:0/4139244298 wait complete. 2024-04-25T14:04:24.175 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T14:04:24.176 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:04:24.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:04:24.252 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T14:04:24.252 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:24.252 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:24.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:04:24.267 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:04:24.267 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:24.310 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:04:24.310 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:24.310 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:24.325 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:04:24.325 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:04:24.325 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:24.366 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:04:24.366 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-25T14:04:24.366 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:24.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.474+0000 7f1dae651640 1 Processor -- start 2024-04-25T14:04:24.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.474+0000 7f1dae651640 1 -- start start 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dae651640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dae651640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1da80b3fe0 0x7f1da8167da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dae651640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1da80d2ee0 con 0x7f1da80b4940 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dae651640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f1da80d1030 con 0x7f1da80b3fe0 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dae651640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1da8161b60 con 0x7f1da806e4d0 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1da77fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1dacbee640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f1da80b3fe0 0x7f1da8167da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1da77fe640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:37892/0 (socket says 172.21.15.150:37892) 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.476+0000 7f1da77fe640 1 -- 172.21.15.150:0/4004349385 learned_addr learned my addr 172.21.15.150:0/4004349385 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.477+0000 7f1da77fe640 1 -- 172.21.15.150:0/4004349385 >> v2:172.21.15.187:3300/0 conn(0x7f1da80b3fe0 msgr2=0x7f1da8167da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.477+0000 7f1da77fe640 1 --2- 172.21.15.150:0/4004349385 >> v2:172.21.15.187:3300/0 conn(0x7f1da80b3fe0 0x7f1da8167da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.477+0000 7f1da77fe640 1 -- 172.21.15.150:0/4004349385 >> v1:172.21.15.187:6789/0 conn(0x7f1da80b4940 legacy=0x7f1da80d25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.477+0000 7f1da77fe640 1 -- 172.21.15.150:0/4004349385 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da80d31c0 con 0x7f1da806e4d0 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.477+0000 7f1da77fe640 1 --2- 172.21.15.150:0/4004349385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009870 tx=0x7f1d9c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad93772a229daf60 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1da6ffd640 1 -- 172.21.15.150:0/4004349385 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c037070 con 0x7f1da806e4d0 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1da6ffd640 1 -- 172.21.15.150:0/4004349385 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d9c032440 con 0x7f1da806e4d0 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1da6ffd640 1 -- 172.21.15.150:0/4004349385 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c032810 con 0x7f1da806e4d0 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 -- 172.21.15.150:0/4004349385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 msgr2=0x7f1da80b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 --2- 172.21.15.150:0/4004349385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c009870 tx=0x7f1d9c02a270 comp rx=0 tx=0).stop 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 -- 172.21.15.150:0/4004349385 shutdown_connections 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 --2- 172.21.15.150:0/4004349385 >> v2:172.21.15.187:3300/0 conn(0x7f1da80b3fe0 0x7f1da8167da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 --2- 172.21.15.150:0/4004349385 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da806e4d0 0x7f1da80b3aa0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 -- 172.21.15.150:0/4004349385 >> 172.21.15.150:0/4004349385 conn(0x7f1da809b4a0 msgr2=0x7f1da809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 -- 172.21.15.150:0/4004349385 shutdown_connections 2024-04-25T14:04:24.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.478+0000 7f1dae651640 1 -- 172.21.15.150:0/4004349385 wait complete. 2024-04-25T14:04:24.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.480+0000 7f1dae651640 1 Processor -- start 2024-04-25T14:04:24.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.480+0000 7f1dae651640 1 -- start start 2024-04-25T14:04:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da80b3fe0 0x7f1da80c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da80b4940 0x7f1da815be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1da80b43f0 con 0x7f1da806e4d0 2024-04-25T14:04:24.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1da806cfa0 con 0x7f1da80b4940 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dae651640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1da80b4570 con 0x7f1da80b3fe0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35108/0 (socket says 172.21.15.150:35108) 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 -- 172.21.15.150:0/4080923995 learned_addr learned my addr 172.21.15.150:0/4080923995 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1dacbee640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da80b4940 0x7f1da815be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da77fe640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da80b3fe0 0x7f1da80c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 -- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da80b3fe0 msgr2=0x7f1da80c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da80b3fe0 0x7f1da80c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 -- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da80b4940 msgr2=0x7f1da815be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da80b4940 0x7f1da815be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 -- 172.21.15.150:0/4080923995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da80c70f0 con 0x7f1da806e4d0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da7fff640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c3c0 tx=0x7f1d9800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.482+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98017070 con 0x7f1da806e4d0 2024-04-25T14:04:24.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.483+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d980052f0 con 0x7f1da806e4d0 2024-04-25T14:04:24.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.483+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da80c7dd0 con 0x7f1da806e4d0 2024-04-25T14:04:24.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.483+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d98003830 con 0x7f1da806e4d0 2024-04-25T14:04:24.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.483+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da80c8390 con 0x7f1da806e4d0 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.484+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da8076a40 con 0x7f1da806e4d0 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.484+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1d9801a070 con 0x7f1da806e4d0 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.485+0000 7f1da4ff9640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 0x7f1d940769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.485+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f1d98094310 con 0x7f1da806e4d0 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.485+0000 7f1da77fe640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 0x7f1d940769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.485+0000 7f1da77fe640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 0x7f1d940769f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c034000 tx=0x7f1d9c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.488 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.489+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1d9805e5b0 con 0x7f1da806e4d0 2024-04-25T14:04:24.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.704+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1da80c8df0 con 0x7f1da806e4d0 2024-04-25T14:04:24.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.704+0000 7f1da4ff9640 1 -- 172.21.15.150:0/4080923995 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+3881 (secure 0 0 0) 0x7f1d980628b0 con 0x7f1da806e4d0 2024-04-25T14:04:24.704 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:24.704 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5036,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1366059233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1366059233},{"type":"v1","addr":"172.21.15.150:6836","nonce":1366059233}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":17},{"gid":5039,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/4102718837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":4102718837},{"type":"v1","addr":"172.21.15.150:6837","nonce":4102718837}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18},{"gid":5371,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/4255006279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":4255006279},{"type":"v1","addr":"172.21.15.187:6834","nonce":4255006279}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":5512,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/1933846539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":1933846539},{"type":"v1","addr":"172.21.15.187:6835","nonce":1933846539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":26}],"filesystems":[]} 2024-04-25T14:04:24.704 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 26 2024-04-25T14:04:24.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 msgr2=0x7f1d940769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 0x7f1d940769f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c034000 tx=0x7f1d9c02e000 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 msgr2=0x7f1da815d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1d9800c3c0 tx=0x7f1d9800c880 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 shutdown_connections 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1d940744e0 0x7f1d940769f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1da80b4940 0x7f1da815be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1da80b3fe0 0x7f1da80c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 --2- 172.21.15.150:0/4080923995 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1da806e4d0 0x7f1da815d3d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 >> 172.21.15.150:0/4080923995 conn(0x7f1da809b4a0 msgr2=0x7f1da80d2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 shutdown_connections 2024-04-25T14:04:24.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.706+0000 7f1dae651640 1 -- 172.21.15.150:0/4080923995 wait complete. 2024-04-25T14:04:24.717 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T14:04:24.717 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T14:04:24.717 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T14:04:24.717 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:24.718 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T14:04:24.718 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T14:04:24.718 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T14:04:24.718 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:24.718 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T14:04:24.718 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T14:04:24.718 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T14:04:24.719 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:24.719 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T14:04:24.719 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T14:04:24.719 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T14:04:24.719 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:24.727 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T14:04:24.727 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T14:04:24.728 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T14:04:24.728 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T14:04:24.731 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T14:04:24.732 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T14:04:24.735 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T14:04:24.735 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T14:04:24.770 INFO:tasks.ceph.mds.a:Started 2024-04-25T14:04:24.770 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:24.771 INFO:tasks.ceph.mds.c:Started 2024-04-25T14:04:24.771 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:24.776 INFO:tasks.ceph.mds.b:Started 2024-04-25T14:04:24.777 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:24.778 INFO:tasks.ceph.mds.d:Started 2024-04-25T14:04:24.778 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:24.778 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-25T14:04:24.855 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:04:24.856 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-04-25T14:04:24.870 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:04:24.871 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:04:24.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.894+0000 7f192ff06640 1 Processor -- start 2024-04-25T14:04:24.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.894+0000 7f192ff06640 1 -- start start 2024-04-25T14:04:24.895 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192ff06640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192ff06640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f19281605d0 0x7f19280d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192ff06640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19280d3570 con 0x7f19280cf8e0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192ff06640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f19280b4800 con 0x7f19280cdfe0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192ff06640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f19280b4ae0 con 0x7f19281605d0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192d4a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192d4a1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35158/0 (socket says 172.21.15.150:35158) 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192d4a1640 1 -- 172.21.15.150:0/860592877 learned_addr learned my addr 172.21.15.150:0/860592877 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192e4a3640 1 --2- 172.21.15.150:0/860592877 >> v2:172.21.15.187:3300/0 conn(0x7f19281605d0 0x7f19280d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192cca0640 1 -- 172.21.15.150:0/860592877 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4186163091 0 0) 0x7f19280d3570 con 0x7f19280cf8e0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192cca0640 1 -- 172.21.15.150:0/860592877 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1914003520 con 0x7f19280cf8e0 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192e4a3640 1 -- 172.21.15.150:0/860592877 >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 msgr2=0x7f19280b4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192e4a3640 1 --2- 172.21.15.150:0/860592877 >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.896 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192e4a3640 1 -- 172.21.15.150:0/860592877 >> v1:172.21.15.187:6789/0 conn(0x7f19280cf8e0 legacy=0x7f19280b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192e4a3640 1 -- 172.21.15.150:0/860592877 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19280b4dc0 con 0x7f19281605d0 2024-04-25T14:04:24.897 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.896+0000 7f192d4a1640 1 --2- 172.21.15.150:0/860592877 >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.897+0000 7f192e4a3640 1 --2- 172.21.15.150:0/860592877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19281605d0 0x7f19280d1ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f19280c6360 tx=0x7f192002a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e697eec0e71b8575 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.897+0000 7f192cca0640 1 -- 172.21.15.150:0/860592877 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920007740 con 0x7f19281605d0 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.897+0000 7f192cca0640 1 -- 172.21.15.150:0/860592877 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19200078e0 con 0x7f19281605d0 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.897+0000 7f192cca0640 1 -- 172.21.15.150:0/860592877 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920007c90 con 0x7f19281605d0 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 -- 172.21.15.150:0/860592877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19281605d0 msgr2=0x7f19280d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 --2- 172.21.15.150:0/860592877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19281605d0 0x7f19280d1ca0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f19280c6360 tx=0x7f192002a680 comp rx=0 tx=0).stop 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 -- 172.21.15.150:0/860592877 shutdown_connections 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 --2- 172.21.15.150:0/860592877 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19281605d0 0x7f19280d1ca0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 --2- 172.21.15.150:0/860592877 >> v2:172.21.15.150:3300/0 conn(0x7f19280cdfe0 0x7f19280b4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 -- 172.21.15.150:0/860592877 >> 172.21.15.150:0/860592877 conn(0x7f192809b4a0 msgr2=0x7f192809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 -- 172.21.15.150:0/860592877 shutdown_connections 2024-04-25T14:04:24.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.898+0000 7f192ff06640 1 -- 172.21.15.150:0/860592877 wait complete. 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.900+0000 7f192ff06640 1 Processor -- start 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.900+0000 7f192ff06640 1 -- start start 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 0x7f1928151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19280cf8e0 0x7f19280cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f19280d2360 con 0x7f19281605d0 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f19280d21e0 con 0x7f19280cdfe0 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192ff06640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f19280d24e0 con 0x7f19280cf8e0 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35168/0 (socket says 172.21.15.150:35168) 2024-04-25T14:04:24.902 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 -- 172.21.15.150:0/37750698 learned_addr learned my addr 172.21.15.150:0/37750698 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192dca2640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 0x7f1928151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192d4a1640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19280cf8e0 0x7f19280cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 -- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19280cf8e0 msgr2=0x7f19280cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19280cf8e0 0x7f19280cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 -- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 msgr2=0x7f1928151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 0x7f1928151d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192e4a3640 1 -- 172.21.15.150:0/37750698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19280ca920 con 0x7f19281605d0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.901+0000 7f192dca2640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 0x7f1928151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f192e4a3640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1920009e50 tx=0x7f1920004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f192003f070 con 0x7f19281605d0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1920031c90 con 0x7f19281605d0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19280c9930 con 0x7f19281605d0 2024-04-25T14:04:24.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1920009450 con 0x7f19281605d0 2024-04-25T14:04:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.902+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19280ca080 con 0x7f19281605d0 2024-04-25T14:04:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.903+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1928076a40 con 0x7f19281605d0 2024-04-25T14:04:24.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.903+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f192003b410 con 0x7f19281605d0 2024-04-25T14:04:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.904+0000 7f191a7fc640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 0x7f190c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.904+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5507+0+0 (secure 0 0 0) 0x7f19200b62e0 con 0x7f19281605d0 2024-04-25T14:04:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.904+0000 7f192dca2640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 0x7f190c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:24.905 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.904+0000 7f192dca2640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 0x7f190c0769f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1924007fd0 tx=0x7f1924009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:24.907 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:24.909+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1920080570 con 0x7f19281605d0 2024-04-25T14:04:25.095 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:25.096+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f19280caaf0 con 0x7f19281605d0 2024-04-25T14:04:26.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.184+0000 7f191a7fc640 1 -- 172.21.15.150:0/37750698 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v52) ==== 97+0+0 (secure 0 0 0) 0x7f1920084870 con 0x7f19281605d0 2024-04-25T14:04:26.183 INFO:teuthology.orchestra.run.smithi150.stderr: removed all blocklist entries 2024-04-25T14:04:26.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 msgr2=0x7f190c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 0x7f190c0769f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1924007fd0 tx=0x7f1924009040 comp rx=0 tx=0).stop 2024-04-25T14:04:26.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 msgr2=0x7f192815dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f1920009e50 tx=0x7f1920004a40 comp rx=0 tx=0).stop 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 shutdown_connections 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f190c0744e0 0x7f190c0769f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f19281605d0 0x7f192815dd30 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f19280cf8e0 0x7f19280cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 --2- 172.21.15.150:0/37750698 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f19280cdfe0 0x7f1928151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 >> 172.21.15.150:0/37750698 conn(0x7f192809b4a0 msgr2=0x7f192815edd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 shutdown_connections 2024-04-25T14:04:26.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.187+0000 7f192ff06640 1 -- 172.21.15.150:0/37750698 wait complete. 2024-04-25T14:04:26.200 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-25T14:04:26.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.314+0000 7fb69e2f9640 1 Processor -- start 2024-04-25T14:04:26.314 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.315+0000 7fb69e2f9640 1 -- start start 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb69e2f9640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb6980b4cd0 0x7fb69806f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb69e2f9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6980b3c30 con 0x7fb6980cfd10 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb69e2f9640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6980b3f10 con 0x7fb6980d22e0 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb69e2f9640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb6980b50e0 con 0x7fb6980b4cd0 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb697fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb6980cfd10 0x7fb6980d1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45598/0 (socket says 172.21.15.150:45598) 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb697fff640 1 -- 172.21.15.150:0/3637130903 learned_addr learned my addr 172.21.15.150:0/3637130903 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:26.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb69c896640 1 --2- 172.21.15.150:0/3637130903 >> v2:172.21.15.187:3300/0 conn(0x7fb6980b4cd0 0x7fb69806f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1384321813 0 0) 0x7fb6980b3c30 con 0x7fb6980cfd10 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.316+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb680003520 con 0x7fb6980cfd10 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 423731109 0 0) 0x7fb680003520 con 0x7fb6980cfd10 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 >> v2:172.21.15.187:3300/0 conn(0x7fb6980b4cd0 msgr2=0x7fb69806f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 --2- 172.21.15.150:0/3637130903 >> v2:172.21.15.187:3300/0 conn(0x7fb6980b4cd0 0x7fb69806f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 >> v1:172.21.15.187:6789/0 conn(0x7fb6980d22e0 legacy=0x7fb6980b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6980b41f0 con 0x7fb6980cfd10 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fb688001670 con 0x7fb6980cfd10 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 >> v1:172.21.15.150:6789/0 conn(0x7fb6980cfd10 legacy=0x7fb6980d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 0x7fb680003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb680004800 0x7fb680004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb680005350 0x7fb68000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb6980b50e0 con 0x7fb6800039b0 2024-04-25T14:04:26.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb68000c280 con 0x7fb680004800 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb68000c480 con 0x7fb680005350 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 0x7fb680003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb6977fe640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb680004800 0x7fb680004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb69c896640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb680005350 0x7fb68000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 -- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb680005350 msgr2=0x7fb68000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb680005350 0x7fb68000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 -- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb680004800 msgr2=0x7fb680004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb680004800 0x7fb680004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb697fff640 1 -- 172.21.15.150:0/3637130903 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb680003520 con 0x7fb6800039b0 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.317+0000 7fb69c896640 1 -- 172.21.15.150:0/3637130903 reap_dead start 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.318+0000 7fb697fff640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 0x7fb680003e20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c210 tx=0x7fb680005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a40a098a8de1a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.318+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688007cf0 con 0x7fb6800039b0 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.318+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb688005af0 con 0x7fb6800039b0 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.318+0000 7fb696ffd640 1 -- 172.21.15.150:0/3637130903 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb688007070 con 0x7fb6800039b0 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 -- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 msgr2=0x7fb680003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 0x7fb680003e20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fb68800c210 tx=0x7fb680005830 comp rx=0 tx=0).stop 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 -- 172.21.15.150:0/3637130903 shutdown_connections 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/3637130903 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6800039b0 0x7fb680003e20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 -- 172.21.15.150:0/3637130903 >> 172.21.15.150:0/3637130903 conn(0x7fb69809b4a0 msgr2=0x7fb69809b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 -- 172.21.15.150:0/3637130903 shutdown_connections 2024-04-25T14:04:26.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.319+0000 7fb69e2f9640 1 -- 172.21.15.150:0/3637130903 wait complete. 2024-04-25T14:04:26.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.321+0000 7fb69e2f9640 1 Processor -- start 2024-04-25T14:04:26.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.321+0000 7fb69e2f9640 1 -- start start 2024-04-25T14:04:26.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 0x7fb6980c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 0x7fb69815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb6980b47b0 con 0x7fb6981faae0 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb6980b39e0 con 0x7fb69815d890 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69e2f9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb69806f500 con 0x7fb6800039b0 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35188/0 (socket says 172.21.15.150:35188) 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 -- 172.21.15.150:0/1631149720 learned_addr learned my addr 172.21.15.150:0/1631149720 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb697fff640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 0x7fb6980c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69c896640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 0x7fb69815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 -- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 msgr2=0x7fb6980c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 0x7fb6980c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 -- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 msgr2=0x7fb69815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 0x7fb69815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb6977fe640 1 -- 172.21.15.150:0/1631149720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69815e1e0 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb697fff640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 0x7fb6980c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.322+0000 7fb69c896640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 0x7fb69815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb6977fe640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00f570 tx=0x7fb68c00fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c0052e0 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb68c018070 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6980c7410 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68c003830 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.323+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6981fa000 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.324+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6980c7690 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.324+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb68c0039d0 con 0x7fb6981faae0 2024-04-25T14:04:26.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.325+0000 7fb694ff9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.325+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7fb68c094d40 con 0x7fb6981faae0 2024-04-25T14:04:26.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.325+0000 7fb697fff640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.325+0000 7fb697fff640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb688032000 tx=0x7fb688007000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.330+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb69815e1e0 con 0x7fb6981faae0 2024-04-25T14:04:26.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.572+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb698076a40 con 0x7fb6981faae0 2024-04-25T14:04:26.571 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.572+0000 7fb694ff9640 1 -- 172.21.15.150:0/1631149720 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) ==== 80+0+5304 (secure 0 0 0) 0x7fb68c05f720 con 0x7fb6981faae0 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "auth_dump": [ 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.a", 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmu4BmJxAA97zMMh3D3KUNtsDbGwkzDA==", 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.572 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.b", 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmwLnMMxAATYxzVsSAZZUjQJuFzeuaSA==", 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.573 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.c", 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmGRrILhAAIg2pUAKfYtkPHonZ2jFm2Q==", 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.574 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mds.d", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmwO4tOxAA8v8nYbz09RgqeKeG1kP8Aw==", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.0", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAqYipma5wlGhAA4H2WY5d5faQLkw54oTdhvg==", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.575 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.1", 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAtYipm9+3RGBAANzSmNtvSkPp3PZqNPOc5hQ==", 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.576 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.2", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAwYipmb72gFxAAXszjhp98sXfU3mERSpXVYA==", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.3", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAzYipmN9yoFhAAUeT/1IDgSofY9Txb6wBIoA==", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.577 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.4", 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA5YipmorY9HhAAStUVEPsdQwhrk3CRCyKReg==", 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.5", 2024-04-25T14:04:26.578 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA8YipmFsVgHRAAuxUDVhP1YYbOIuqKnk/0rg==", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.6", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQA/YipmCeXsGxAAMKhPXK0qkbkSvZjER5hCxA==", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.579 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "osd.7", 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQBCYipmfInFGhAAQzv1lzLJbEZsVlzxNER6xw==", 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow profile osd", 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile osd", 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.580 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.0", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ==", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow r", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.1", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmDUJACxAAcFyylAXclzW4D6C20RFk4Q==", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:04:26.581 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.2", 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipm2NSoBxAABm1fUKNHFMNuiHMmSMrKlA==", 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:04:26.582 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.3", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAnYipmm2fQDhAANojc8uxJB15T5dVpgYA4Vg==", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow r", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow rw", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow rwx" 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "client.admin", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmNaqiCRAANv+ub/K0QsBwYm9UTT+2Fg==", 2024-04-25T14:04:26.583 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "mgr": "allow *", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow *", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.x", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmoMzCIxAAMQ8RkeGQMgn4UuygerC8eg==", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.584 INFO:teuthology.orchestra.run.smithi150.stdout: }, 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "entity": "mgr.y", 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "key": "AQAmYipmdMg8HhAAGiDa9V7HIfyPF1HvP9u4rQ==", 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "caps": { 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "mds": "allow *", 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "mon": "allow profile mgr", 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: "osd": "allow *" 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout: ] 2024-04-25T14:04:26.585 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 msgr2=0x7fb6840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb688032000 tx=0x7fb688007000 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 msgr2=0x7fb6980a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fb68c00f570 tx=0x7fb68c00fa30 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 shutdown_connections 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb6840744b0 0x7fb6840769c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb69815d890 0x7fb69815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb6981faae0 0x7fb6980a0180 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 --2- 172.21.15.150:0/1631149720 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb6800039b0 0x7fb6980c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.587 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 >> 172.21.15.150:0/1631149720 conn(0x7fb69809b4a0 msgr2=0x7fb698071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:26.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 shutdown_connections 2024-04-25T14:04:26.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.574+0000 7fb69e2f9640 1 -- 172.21.15.150:0/1631149720 wait complete. 2024-04-25T14:04:26.588 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:26.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.697+0000 7f591d5b4640 1 Processor -- start 2024-04-25T14:04:26.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.697+0000 7f591d5b4640 1 -- start start 2024-04-25T14:04:26.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f591d5b4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59180ae7e0 0x7f59180aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f591d5b4640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59180b4750 con 0x7f59180b57f0 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f591d5b4640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59180b4a30 con 0x7f591815df00 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f591d5b4640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f591815e310 con 0x7f59180ae7e0 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59180ae7e0 0x7f59180aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f59180ae7e0 0x7f59180aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35204/0 (socket says 172.21.15.150:35204) 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59167fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f59180b57f0 0x7f59180b5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:45604/0 (socket says 172.21.15.150:45604) 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 -- 172.21.15.150:0/961596565 learned_addr learned my addr 172.21.15.150:0/961596565 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 -- 172.21.15.150:0/961596565 >> v1:172.21.15.187:6789/0 conn(0x7f591815df00 legacy=0x7f591806f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 -- 172.21.15.150:0/961596565 >> v1:172.21.15.150:6789/0 conn(0x7f59180b57f0 legacy=0x7f59180b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f59177fe640 1 -- 172.21.15.150:0/961596565 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59180b4d10 con 0x7f59180ae7e0 2024-04-25T14:04:26.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.699+0000 7f5915ffb640 1 -- 172.21.15.150:0/961596565 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273521622 0 0) 0x7f59180b4750 con 0x7f59180b57f0 2024-04-25T14:04:26.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.700+0000 7f59177fe640 1 --2- 172.21.15.150:0/961596565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f59180aebf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5910009e50 tx=0x7f591002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b44b4e5026da5940 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.700+0000 7f5915ffb640 1 -- 172.21.15.150:0/961596565 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5910036070 con 0x7f59180ae7e0 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.700+0000 7f5915ffb640 1 -- 172.21.15.150:0/961596565 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5910031600 con 0x7f59180ae7e0 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.700+0000 7f5915ffb640 1 -- 172.21.15.150:0/961596565 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59100319d0 con 0x7f59180ae7e0 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 -- 172.21.15.150:0/961596565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 msgr2=0x7f59180aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 --2- 172.21.15.150:0/961596565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f59180aebf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f5910009e50 tx=0x7f591002a730 comp rx=0 tx=0).stop 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 -- 172.21.15.150:0/961596565 shutdown_connections 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 --2- 172.21.15.150:0/961596565 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f59180aebf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 -- 172.21.15.150:0/961596565 >> 172.21.15.150:0/961596565 conn(0x7f59180d1ae0 msgr2=0x7f59180d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 -- 172.21.15.150:0/961596565 shutdown_connections 2024-04-25T14:04:26.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.701+0000 7f591d5b4640 1 -- 172.21.15.150:0/961596565 wait complete. 2024-04-25T14:04:26.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.703+0000 7f591d5b4640 1 Processor -- start 2024-04-25T14:04:26.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.703+0000 7f591d5b4640 1 -- start start 2024-04-25T14:04:26.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 0x7f59181f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 0x7f59180b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f591806f950 con 0x7f59180ae7e0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f59180b5360 con 0x7f59180b57f0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f591d5b4640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f59180af130 con 0x7f591815df00 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f5916ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f5916ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35214/0 (socket says 172.21.15.150:35214) 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.704+0000 7f5916ffd640 1 -- 172.21.15.150:0/1637260851 learned_addr learned my addr 172.21.15.150:0/1637260851 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f59167fc640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 0x7f59181f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f59177fe640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 0x7f59180b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 -- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 msgr2=0x7f59180b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 0x7f59180b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 -- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 msgr2=0x7f59181f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 0x7f59181f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 -- 172.21.15.150:0/1637260851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59180ce010 con 0x7f59180ae7e0 2024-04-25T14:04:26.705 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f59167fc640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 0x7f59181f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f59177fe640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 0x7f59180b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f5916ffd640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f590800c400 tx=0x7f590800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908017070 con 0x7f59180ae7e0 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59080052d0 con 0x7f59180ae7e0 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59180ce2f0 con 0x7f59180ae7e0 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5908003830 con 0x7f59180ae7e0 2024-04-25T14:04:26.706 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.705+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59180ce8b0 con 0x7f59180ae7e0 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.706+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5918076a40 con 0x7f59180ae7e0 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.707+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f590801a070 con 0x7f59180ae7e0 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.707+0000 7f58ff7fe640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 0x7f59040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.707+0000 7f59167fc640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 0x7f59040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.707+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f59080944b0 con 0x7f59180ae7e0 2024-04-25T14:04:26.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.708+0000 7f59167fc640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 0x7f59040769c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f59180b2830 tx=0x7f590c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:26.710 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.712+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f590805ed80 con 0x7f59180ae7e0 2024-04-25T14:04:26.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.887+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f59180710a0 con 0x7f59180ae7e0 2024-04-25T14:04:26.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.888+0000 7f58ff7fe640 1 -- 172.21.15.150:0/1637260851 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) ==== 74+0+13164 (secure 0 0 0) 0x7f590805e680 con 0x7f59180ae7e0 2024-04-25T14:04:26.888 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:26.888 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":52,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:26.179939+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 msgr2=0x7f59040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 0x7f59040769c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f59180b2830 tx=0x7f590c009040 comp rx=0 tx=0).stop 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 msgr2=0x7f591814f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f590800c400 tx=0x7f590800c8c0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 shutdown_connections 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f59040744b0 0x7f59040769c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f591815df00 0x7f59180b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f59180b57f0 0x7f59181f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 --2- 172.21.15.150:0/1637260851 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59180ae7e0 0x7f591814f0e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:26.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.890+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 >> 172.21.15.150:0/1637260851 conn(0x7f59180d1ae0 msgr2=0x7f5918153bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:26.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.891+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 shutdown_connections 2024-04-25T14:04:26.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:26.891+0000 7f591d5b4640 1 -- 172.21.15.150:0/1637260851 wait complete. 2024-04-25T14:04:26.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:04:26.902 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:04:27.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.012+0000 7fd1cb296640 1 Processor -- start 2024-04-25T14:04:27.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.012+0000 7fd1cb296640 1 -- start start 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.013+0000 7fd1cb296640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 0x7fd1c406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.013+0000 7fd1cb296640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.013+0000 7fd1cb296640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c406ff50 con 0x7fd1c40b5900 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.013+0000 7fd1cb296640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c40b5d10 con 0x7fd1c4163d80 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.013+0000 7fd1cb296640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c406f3d0 con 0x7fd1c4160460 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c8831640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c8831640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:45266/0 (socket says 172.21.15.150:45266) 2024-04-25T14:04:27.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9833640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fd1c40b5900 0x7fd1c4072f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:48628/0 (socket says 172.21.15.150:48628) 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c8831640 1 -- 172.21.15.150:0/2371763732 learned_addr learned my addr 172.21.15.150:0/2371763732 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9032640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 0x7fd1c406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9032640 1 -- 172.21.15.150:0/2371763732 >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 msgr2=0x7fd1c40b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9032640 1 --2- 172.21.15.150:0/2371763732 >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9032640 1 -- 172.21.15.150:0/2371763732 >> v1:172.21.15.187:6789/0 conn(0x7fd1c40b5900 legacy=0x7fd1c4072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1bbfff640 1 -- 172.21.15.150:0/2371763732 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750411288 0 0) 0x7fd1c406ff50 con 0x7fd1c40b5900 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c9032640 1 -- 172.21.15.150:0/2371763732 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c406f6b0 con 0x7fd1c4160460 2024-04-25T14:04:27.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.014+0000 7fd1c8831640 1 --2- 172.21.15.150:0/2371763732 >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:27.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.015+0000 7fd1c9032640 1 --2- 172.21.15.150:0/2371763732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 0x7fd1c406ed40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00097a0 tx=0x7fd1c002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30e36c6202342fc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.015+0000 7fd1bbfff640 1 -- 172.21.15.150:0/2371763732 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0037070 con 0x7fd1c4160460 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.015+0000 7fd1bbfff640 1 -- 172.21.15.150:0/2371763732 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c002ae50 con 0x7fd1c4160460 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.015+0000 7fd1bbfff640 1 -- 172.21.15.150:0/2371763732 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c0032660 con 0x7fd1c4160460 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 -- 172.21.15.150:0/2371763732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 msgr2=0x7fd1c406ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2371763732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 0x7fd1c406ed40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c00097a0 tx=0x7fd1c002a100 comp rx=0 tx=0).stop 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 -- 172.21.15.150:0/2371763732 shutdown_connections 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2371763732 >> v2:172.21.15.187:3300/0 conn(0x7fd1c4163d80 0x7fd1c40b53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2371763732 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4160460 0x7fd1c406ed40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 -- 172.21.15.150:0/2371763732 >> 172.21.15.150:0/2371763732 conn(0x7fd1c40d03c0 msgr2=0x7fd1c40d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 -- 172.21.15.150:0/2371763732 shutdown_connections 2024-04-25T14:04:27.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.016+0000 7fd1cb296640 1 -- 172.21.15.150:0/2371763732 wait complete. 2024-04-25T14:04:27.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.017+0000 7fd1cb296640 1 Processor -- start 2024-04-25T14:04:27.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.017+0000 7fd1cb296640 1 -- start start 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40b5900 0x7fd1c4159730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4163d80 0x7fd1c40ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c40b3030 con 0x7fd1c40cd130 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c406fc70 con 0x7fd1c40b5900 2024-04-25T14:04:27.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1cb296640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c40ce5c0 con 0x7fd1c4163d80 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35226/0 (socket says 172.21.15.150:35226) 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 -- 172.21.15.150:0/2957393785 learned_addr learned my addr 172.21.15.150:0/2957393785 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c8831640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4163d80 0x7fd1c40ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9032640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40b5900 0x7fd1c4159730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 -- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4163d80 msgr2=0x7fd1c40ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4163d80 0x7fd1c40ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 -- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40b5900 msgr2=0x7fd1c4159730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40b5900 0x7fd1c4159730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 -- 172.21.15.150:0/2957393785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c41fa2d0 con 0x7fd1c40cd130 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.019+0000 7fd1c9833640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c780 tx=0x7fd1bc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.020+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc018070 con 0x7fd1c40cd130 2024-04-25T14:04:27.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.020+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1bc00ce30 con 0x7fd1c40cd130 2024-04-25T14:04:27.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.020+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c41f9510 con 0x7fd1c40cd130 2024-04-25T14:04:27.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.020+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1bc00a860 con 0x7fd1c40cd130 2024-04-25T14:04:27.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.020+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c41f9ad0 con 0x7fd1c40cd130 2024-04-25T14:04:27.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.021+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1c4076a40 con 0x7fd1c40cd130 2024-04-25T14:04:27.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.021+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd1bc01b070 con 0x7fd1c40cd130 2024-04-25T14:04:27.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.022+0000 7fd1b9ffb640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 0x7fd1a80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.022+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7fd1bc0944c0 con 0x7fd1c40cd130 2024-04-25T14:04:27.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.022+0000 7fd1c9032640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 0x7fd1a80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.022+0000 7fd1c9032640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 0x7fd1a80769c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0034000 tx=0x7fd1c002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.027+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1bc05e7b0 con 0x7fd1c40cd130 2024-04-25T14:04:27.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.202+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1c40cd5e0 con 0x7fd1c40cd130 2024-04-25T14:04:27.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.202+0000 7fd1b9ffb640 1 -- 172.21.15.150:0/2957393785 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) ==== 93+0+10 (secure 0 0 0) 0x7fd1bc062ab0 con 0x7fd1c40cd130 2024-04-25T14:04:27.202 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 msgr2=0x7fd1a80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 0x7fd1a80769c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0034000 tx=0x7fd1c002e000 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 msgr2=0x7fd1c40b4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd1bc00c780 tx=0x7fd1bc00cc40 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 shutdown_connections 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd1a80744b0 0x7fd1a80769c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd1c40cd130 0x7fd1c40b4990 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd1c4163d80 0x7fd1c40ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 --2- 172.21.15.150:0/2957393785 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd1c40b5900 0x7fd1c4159730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.204+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 >> 172.21.15.150:0/2957393785 conn(0x7fd1c40d03c0 msgr2=0x7fd1c415a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:27.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.205+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 shutdown_connections 2024-04-25T14:04:27.205 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.205+0000 7fd1cb296640 1 -- 172.21.15.150:0/2957393785 wait complete. 2024-04-25T14:04:27.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T14:04:27.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.324+0000 7f00a740d640 1 Processor -- start 2024-04-25T14:04:27.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.324+0000 7f00a740d640 1 -- start start 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.325+0000 7f00a740d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.325+0000 7f00a740d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00a00b5f00 0x7f00a00ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.325+0000 7f00a740d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00a00b4750 con 0x7f00a006f780 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.325+0000 7f00a740d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f00a015e310 con 0x7f00a00b5f00 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.325+0000 7f00a740d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00a00b5360 con 0x7f00a015df00 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a51a9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00a00b5f00 0x7f00a00ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f00a00b5f00 0x7f00a00ae7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:45280/0 (socket says 172.21.15.150:45280) 2024-04-25T14:04:27.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a51a9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:37990/0 (socket says 172.21.15.150:37990) 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 -- 172.21.15.150:0/1470903042 learned_addr learned my addr 172.21.15.150:0/1470903042 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a49a8640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f00a006f780 0x7f00a00b57f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:48642/0 (socket says 172.21.15.150:48642) 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 -- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 msgr2=0x7f00a006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 -- 172.21.15.150:0/1470903042 >> v1:172.21.15.187:6789/0 conn(0x7f00a006f780 legacy=0x7f00a00b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a59aa640 1 -- 172.21.15.150:0/1470903042 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a00b4a30 con 0x7f00a00b5f00 2024-04-25T14:04:27.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.326+0000 7f00a51a9640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:27.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.327+0000 7f00a59aa640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a00b5f00 0x7f00a00ae7e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f009800d150 tx=0x7f009802c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9e28c444946b73a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.327+0000 7f0097fff640 1 -- 172.21.15.150:0/1470903042 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0098039070 con 0x7f00a00b5f00 2024-04-25T14:04:27.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.327+0000 7f0097fff640 1 -- 172.21.15.150:0/1470903042 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0098009600 con 0x7f00a00b5f00 2024-04-25T14:04:27.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.327+0000 7f0097fff640 1 -- 172.21.15.150:0/1470903042 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00980099d0 con 0x7f00a00b5f00 2024-04-25T14:04:27.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 -- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a00b5f00 msgr2=0x7f00a00ae7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a00b5f00 0x7f00a00ae7e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f009800d150 tx=0x7f009802c890 comp rx=0 tx=0).stop 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 -- 172.21.15.150:0/1470903042 shutdown_connections 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a00b5f00 0x7f00a00ae7e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 --2- 172.21.15.150:0/1470903042 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a015df00 0x7f00a006f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 -- 172.21.15.150:0/1470903042 >> 172.21.15.150:0/1470903042 conn(0x7f00a00d1ae0 msgr2=0x7f00a00d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 -- 172.21.15.150:0/1470903042 shutdown_connections 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.328+0000 7f00a740d640 1 -- 172.21.15.150:0/1470903042 wait complete. 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.329+0000 7f00a740d640 1 Processor -- start 2024-04-25T14:04:27.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.330+0000 7f00a740d640 1 -- start start 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a00b5f00 0x7f00a01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a015df00 0x7f00a01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00aed20 con 0x7f00a006f780 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f00a00b4ff0 con 0x7f00a015df00 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a740d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f00a00aeea0 con 0x7f00a00b5f00 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:35242/0 (socket says 172.21.15.150:35242) 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 -- 172.21.15.150:0/3508955348 learned_addr learned my addr 172.21.15.150:0/3508955348 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a59aa640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a015df00 0x7f00a01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a49a8640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a00b5f00 0x7f00a01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 -- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a00b5f00 msgr2=0x7f00a01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a00b5f00 0x7f00a01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 -- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a015df00 msgr2=0x7f00a01579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a015df00 0x7f00a01579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 -- 172.21.15.150:0/3508955348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00a01f3020 con 0x7f00a006f780 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.331+0000 7f00a51a9640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f009c00e960 tx=0x7f009c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.332 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.332+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c017070 con 0x7f00a006f780 2024-04-25T14:04:27.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.332+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009c00ca00 con 0x7f00a006f780 2024-04-25T14:04:27.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.332+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00a01f2d40 con 0x7f00a006f780 2024-04-25T14:04:27.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.332+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009c00cdd0 con 0x7f00a006f780 2024-04-25T14:04:27.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.332+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00a01e59b0 con 0x7f00a006f780 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.333+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00a00af990 con 0x7f00a006f780 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.333+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f009c01a070 con 0x7f00a006f780 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.334+0000 7f0095ffb640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 0x7f00840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.334+0000 7f00a49a8640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 0x7f00840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.334+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f009c094e00 con 0x7f00a006f780 2024-04-25T14:04:27.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.335+0000 7f00a49a8640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 0x7f00840769c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00a0071330 tx=0x7f0090007e80 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.339+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00a00af990 con 0x7f00a006f780 2024-04-25T14:04:27.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.516+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f00a00aff50 con 0x7f00a006f780 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.517+0000 7f0095ffb640 1 -- 172.21.15.150:0/3508955348 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v52) ==== 84+0+65 (secure 0 0 0) 0x7f00a00aff50 con 0x7f00a006f780 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stdout:[ 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stdout: { 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stdout: "poolnum": 1, 2024-04-25T14:04:27.516 INFO:teuthology.orchestra.run.smithi150.stdout: "poolname": ".mgr" 2024-04-25T14:04:27.517 INFO:teuthology.orchestra.run.smithi150.stdout: } 2024-04-25T14:04:27.517 INFO:teuthology.orchestra.run.smithi150.stdout:] 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 msgr2=0x7f00840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 0x7f00840769c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f00a0071330 tx=0x7f0090007e80 comp rx=0 tx=0).stop 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 msgr2=0x7f00a0151390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f009c00e960 tx=0x7f009c00ee20 comp rx=0 tx=0).stop 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 shutdown_connections 2024-04-25T14:04:27.518 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f00840744b0 0x7f00840769c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f00a015df00 0x7f00a01579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f00a00b5f00 0x7f00a01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 --2- 172.21.15.150:0/3508955348 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f00a006f780 0x7f00a0151390 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 >> 172.21.15.150:0/3508955348 conn(0x7f00a00d1ae0 msgr2=0x7f00a006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 shutdown_connections 2024-04-25T14:04:27.519 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.519+0000 7f00a740d640 1 -- 172.21.15.150:0/3508955348 wait complete. 2024-04-25T14:04:27.529 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T14:04:27.529 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T14:04:27.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.642+0000 7f04f3a27640 1 Processor -- start 2024-04-25T14:04:27.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.642+0000 7f04f3a27640 1 -- start start 2024-04-25T14:04:27.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.643+0000 7f04f3a27640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04ec0b5b30 0x7f04ec0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.643+0000 7f04f3a27640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 0x7f04ec163eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.643+0000 7f04f3a27640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04ec160a10 con 0x7f04ec06ec00 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.643+0000 7f04f3a27640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f04ec1607b0 con 0x7f04ec0b5b30 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.643+0000 7f04f3a27640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec160cb0 con 0x7f04ec0d1800 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f0fc2640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f04ec0b5b30 0x7f04ec0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f1fc4640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 0x7f04ec163eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f17c3640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7f04ec06ec00 0x7f04ec0b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:56272/0 (socket says 172.21.15.150:56272) 2024-04-25T14:04:27.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f17c3640 1 -- 172.21.15.150:0/3545537595 learned_addr learned my addr 172.21.15.150:0/3545537595 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f1fc4640 1 -- 172.21.15.150:0/3545537595 >> v2:172.21.15.187:3300/0 conn(0x7f04ec0b5b30 msgr2=0x7f04ec0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f1fc4640 1 --2- 172.21.15.150:0/3545537595 >> v2:172.21.15.187:3300/0 conn(0x7f04ec0b5b30 0x7f04ec0d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f1fc4640 1 -- 172.21.15.150:0/3545537595 >> v1:172.21.15.187:6789/0 conn(0x7f04ec06ec00 legacy=0x7f04ec0b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.644+0000 7f04f1fc4640 1 -- 172.21.15.150:0/3545537595 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec160f90 con 0x7f04ec0d1800 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.645+0000 7f04f1fc4640 1 --2- 172.21.15.150:0/3545537595 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 0x7f04ec163eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96f371df35a6af7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.645+0000 7f04e3fff640 1 -- 172.21.15.150:0/3545537595 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4037070 con 0x7f04ec0d1800 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.645+0000 7f04e3fff640 1 -- 172.21.15.150:0/3545537595 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04e4032440 con 0x7f04ec0d1800 2024-04-25T14:04:27.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.645+0000 7f04e3fff640 1 -- 172.21.15.150:0/3545537595 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04e4032810 con 0x7f04ec0d1800 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 -- 172.21.15.150:0/3545537595 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 msgr2=0x7f04ec163eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 --2- 172.21.15.150:0/3545537595 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 0x7f04ec163eb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402a180 comp rx=0 tx=0).stop 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 -- 172.21.15.150:0/3545537595 shutdown_connections 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 --2- 172.21.15.150:0/3545537595 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0d1800 0x7f04ec163eb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 --2- 172.21.15.150:0/3545537595 >> v2:172.21.15.187:3300/0 conn(0x7f04ec0b5b30 0x7f04ec0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 -- 172.21.15.150:0/3545537595 >> 172.21.15.150:0/3545537595 conn(0x7f04ec0d0420 msgr2=0x7f04ec0d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:27.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 -- 172.21.15.150:0/3545537595 shutdown_connections 2024-04-25T14:04:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.646+0000 7f04f3a27640 1 -- 172.21.15.150:0/3545537595 wait complete. 2024-04-25T14:04:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.647+0000 7f04f3a27640 1 Processor -- start 2024-04-25T14:04:27.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.647+0000 7f04f3a27640 1 -- start start 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04ec06ec00 0x7f04ec0cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0af310 0x7f04ec0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec06fd70 con 0x7f04ec0d1800 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f04ec161550 con 0x7f04ec06ec00 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f3a27640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f04ec06fef0 con 0x7f04ec0af310 2024-04-25T14:04:27.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59826/0 (socket says 172.21.15.150:59826) 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 -- 172.21.15.150:0/2935607114 learned_addr learned my addr 172.21.15.150:0/2935607114 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f1fc4640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0af310 0x7f04ec0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f17c3640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04ec06ec00 0x7f04ec0cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 -- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0af310 msgr2=0x7f04ec0b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0af310 0x7f04ec0b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 -- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04ec06ec00 msgr2=0x7f04ec0cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04ec06ec00 0x7f04ec0cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 -- 172.21.15.150:0/2935607114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04ec0cbf70 con 0x7f04ec0d1800 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04f0fc2640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e8a0 tx=0x7f04dc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.649+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc017070 con 0x7f04ec0d1800 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.650+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04dc00c940 con 0x7f04ec0d1800 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.650+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04ec0cc3f0 con 0x7f04ec0d1800 2024-04-25T14:04:27.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.650+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04dc00cd30 con 0x7f04ec0d1800 2024-04-25T14:04:27.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.650+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04ec0b6180 con 0x7f04ec0d1800 2024-04-25T14:04:27.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.651+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec0cc140 con 0x7f04ec0d1800 2024-04-25T14:04:27.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.651+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f04dc01a070 con 0x7f04ec0d1800 2024-04-25T14:04:27.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.652+0000 7f04e1ffb640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 0x7f04d00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:27.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.652+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(52..52 src has 1..52) ==== 5249+0+0 (secure 0 0 0) 0x7f04dc094d90 con 0x7f04ec0d1800 2024-04-25T14:04:27.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.652+0000 7f04f17c3640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 0x7f04d00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:27.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.652+0000 7f04f17c3640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 0x7f04d00769f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f04e80012e0 tx=0x7f04e8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:27.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.657+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04dc05f120 con 0x7f04ec0d1800 2024-04-25T14:04:27.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:27.832+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f04ec076a40 con 0x7f04ec0d1800 2024-04-25T14:04:28.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.187+0000 7f04e1ffb640 1 -- 172.21.15.150:0/2935607114 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v53) ==== 138+0+0 (secure 0 0 0) 0x7f04dc063420 con 0x7f04ec0d1800 2024-04-25T14:04:28.186 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' created 2024-04-25T14:04:28.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 msgr2=0x7f04d00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 0x7f04d00769f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f04e80012e0 tx=0x7f04e8009040 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 msgr2=0x7f04ec15eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00e8a0 tx=0x7f04dc00ed60 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 shutdown_connections 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04d00744e0 0x7f04d00769f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f04ec0af310 0x7f04ec0b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04ec0d1800 0x7f04ec15eeb0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 --2- 172.21.15.150:0/2935607114 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f04ec06ec00 0x7f04ec0cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 >> 172.21.15.150:0/2935607114 conn(0x7f04ec0d0420 msgr2=0x7f04ec1624d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 shutdown_connections 2024-04-25T14:04:28.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.189+0000 7f04f3a27640 1 -- 172.21.15.150:0/2935607114 wait complete. 2024-04-25T14:04:28.203 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T14:04:28.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.318+0000 7fa440700640 1 Processor -- start 2024-04-25T14:04:28.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.318+0000 7fa440700640 1 -- start start 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa440700640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 0x7fa43815dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa440700640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa440700640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4380cc5f0 con 0x7fa43806daa0 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa440700640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa43806deb0 con 0x7fa4380cd140 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa440700640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa43806d610 con 0x7fa438162540 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43e49c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa43806daa0 0x7fa4380cca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41160/0 (socket says 172.21.15.150:41160) 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43e49c640 1 -- 172.21.15.150:0/1584846107 learned_addr learned my addr 172.21.15.150:0/1584846107 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/1584846107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43dc9b640 1 --2- 172.21.15.150:0/1584846107 >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 0x7fa43815dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1060000146 0 0) 0x7fa4380cc5f0 con 0x7fa43806daa0 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.320+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa424003520 con 0x7fa43806daa0 2024-04-25T14:04:28.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43ec9d640 1 -- 172.21.15.150:0/1584846107 >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 msgr2=0x7fa43815dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:28.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/1584846107 >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 0x7fa43815dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 72725537 0 0) 0x7fa424003520 con 0x7fa43806daa0 2024-04-25T14:04:28.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43ec9d640 1 -- 172.21.15.150:0/1584846107 >> v1:172.21.15.150:6789/0 conn(0x7fa43806daa0 legacy=0x7fa4380cca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43ec9d640 1 -- 172.21.15.150:0/1584846107 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43806ca00 con 0x7fa438162540 2024-04-25T14:04:28.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43dc9b640 1 --2- 172.21.15.150:0/1584846107 >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 0x7fa43815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:28.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.321+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/1584846107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380af040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4300097a0 tx=0x7fa43002a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e9679b6b7c9f00e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:28.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430037070 con 0x7fa438162540 2024-04-25T14:04:28.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43002aba0 con 0x7fa438162540 2024-04-25T14:04:28.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa43d49a640 1 -- 172.21.15.150:0/1584846107 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4340019f0 con 0x7fa438162540 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 -- 172.21.15.150:0/1584846107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 msgr2=0x7fa4380af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 --2- 172.21.15.150:0/1584846107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380af040 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa4300097a0 tx=0x7fa43002a110 comp rx=0 tx=0).stop 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 -- 172.21.15.150:0/1584846107 shutdown_connections 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 --2- 172.21.15.150:0/1584846107 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380af040 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 --2- 172.21.15.150:0/1584846107 >> v2:172.21.15.187:3300/0 conn(0x7fa4380cd140 0x7fa43815dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.322+0000 7fa440700640 1 -- 172.21.15.150:0/1584846107 >> 172.21.15.150:0/1584846107 conn(0x7fa4380ce390 msgr2=0x7fa4380ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.323+0000 7fa440700640 1 -- 172.21.15.150:0/1584846107 shutdown_connections 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.323+0000 7fa440700640 1 -- 172.21.15.150:0/1584846107 wait complete. 2024-04-25T14:04:28.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.324+0000 7fa440700640 1 Processor -- start 2024-04-25T14:04:28.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.324+0000 7fa440700640 1 -- start start 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43806daa0 0x7fa438144890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa43806d2a0 con 0x7fa4381470d0 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa43806cfc0 con 0x7fa43806daa0 2024-04-25T14:04:28.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa440700640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa438162950 con 0x7fa438162540 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59834/0 (socket says 172.21.15.150:59834) 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 -- 172.21.15.150:0/3861322623 learned_addr learned my addr 172.21.15.150:0/3861322623 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43dc9b640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43e49c640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43806daa0 0x7fa438144890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 -- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 msgr2=0x7fa4380b6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380b6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 -- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43806daa0 msgr2=0x7fa438144890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43806daa0 0x7fa438144890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 -- 172.21.15.150:0/3861322623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa438144dd0 con 0x7fa4381470d0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa43ec9d640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa430035040 tx=0x7fa4300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430037050 con 0x7fa4381470d0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.326+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa430043070 con 0x7fa4381470d0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.327+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4380b6fa0 con 0x7fa4381470d0 2024-04-25T14:04:28.327 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.327+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa430006e10 con 0x7fa4381470d0 2024-04-25T14:04:28.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.327+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4380b5ca0 con 0x7fa4381470d0 2024-04-25T14:04:28.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.328+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa438162360 con 0x7fa4381470d0 2024-04-25T14:04:28.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.328+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa43003ca50 con 0x7fa4381470d0 2024-04-25T14:04:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.329+0000 7fa42effd640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.329+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5619+0+0 (secure 0 0 0) 0x7fa4300b5b30 con 0x7fa4381470d0 2024-04-25T14:04:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.329+0000 7fa43e49c640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:28.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.329+0000 7fa43e49c640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 0x7fa41c0769c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa434001560 tx=0x7fa434009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:28.333 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.334+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa438162360 con 0x7fa4381470d0 2024-04-25T14:04:28.508 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:28.509+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa4380cfe10 con 0x7fa4381470d0 2024-04-25T14:04:29.192 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.193+0000 7fa42effd640 1 -- 172.21.15.150:0/3861322623 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v54) ==== 170+0+0 (secure 0 0 0) 0x7fa4300802f0 con 0x7fa4381470d0 2024-04-25T14:04:29.192 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' created 2024-04-25T14:04:29.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 msgr2=0x7fa41c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 0x7fa41c0769c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa434001560 tx=0x7fa434009040 comp rx=0 tx=0).stop 2024-04-25T14:04:29.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 msgr2=0x7fa4381474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.195 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa430035040 tx=0x7fa4300046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 shutdown_connections 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa41c0744b0 0x7fa41c0769c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa4381470d0 0x7fa4381474e0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa438162540 0x7fa4380b6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 --2- 172.21.15.150:0/3861322623 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa43806daa0 0x7fa438144890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 >> 172.21.15.150:0/3861322623 conn(0x7fa4380ce390 msgr2=0x7fa4380cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 shutdown_connections 2024-04-25T14:04:29.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.196+0000 7fa440700640 1 -- 172.21.15.150:0/3861322623 wait complete. 2024-04-25T14:04:29.219 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T14:04:29.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.336+0000 7fd7a17f3640 1 Processor -- start 2024-04-25T14:04:29.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.336+0000 7fd7a17f3640 1 -- start start 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd7a17f3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd79c160150 0x7fd79c0b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd7a17f3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd79c06d800 0x7fd79c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd7a17f3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd79c0d4f20 con 0x7fd79c0b4c10 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd7a17f3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd79c06cc60 con 0x7fd79c160150 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd7a17f3640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd79c160560 con 0x7fd79c06d800 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd79c160150 0x7fd79c0b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd79affd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd79c0b4c10 0x7fd79c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41166/0 (socket says 172.21.15.150:41166) 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd79c160150 0x7fd79c0b91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59842/0 (socket says 172.21.15.150:59842) 2024-04-25T14:04:29.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd79affd640 1 -- 172.21.15.150:0/2249324999 learned_addr learned my addr 172.21.15.150:0/2249324999 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd79b7fe640 1 --2- 172.21.15.150:0/2249324999 >> v2:172.21.15.187:3300/0 conn(0x7fd79c06d800 0x7fd79c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 -- 172.21.15.150:0/2249324999 >> v2:172.21.15.187:3300/0 conn(0x7fd79c06d800 msgr2=0x7fd79c0d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 --2- 172.21.15.150:0/2249324999 >> v2:172.21.15.187:3300/0 conn(0x7fd79c06d800 0x7fd79c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 -- 172.21.15.150:0/2249324999 >> v1:172.21.15.150:6789/0 conn(0x7fd79c0b4c10 legacy=0x7fd79c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.338+0000 7fd793fff640 1 -- 172.21.15.150:0/2249324999 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c0d5200 con 0x7fd79c160150 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.339+0000 7fd793fff640 1 --2- 172.21.15.150:0/2249324999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c160150 0x7fd79c0b91f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0d0690 tx=0x7fd78402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7527484f775d55e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.339+0000 7fd79a7fc640 1 -- 172.21.15.150:0/2249324999 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784037070 con 0x7fd79c160150 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.339+0000 7fd79a7fc640 1 -- 172.21.15.150:0/2249324999 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd784032440 con 0x7fd79c160150 2024-04-25T14:04:29.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.339+0000 7fd79a7fc640 1 -- 172.21.15.150:0/2249324999 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784032810 con 0x7fd79c160150 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 -- 172.21.15.150:0/2249324999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c160150 msgr2=0x7fd79c0b91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/2249324999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c160150 0x7fd79c0b91f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fd79c0d0690 tx=0x7fd78402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 -- 172.21.15.150:0/2249324999 shutdown_connections 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/2249324999 >> v2:172.21.15.187:3300/0 conn(0x7fd79c06d800 0x7fd79c0d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/2249324999 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c160150 0x7fd79c0b91f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 -- 172.21.15.150:0/2249324999 >> 172.21.15.150:0/2249324999 conn(0x7fd79c0b7690 msgr2=0x7fd79c0b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:29.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 -- 172.21.15.150:0/2249324999 shutdown_connections 2024-04-25T14:04:29.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.340+0000 7fd7a17f3640 1 -- 172.21.15.150:0/2249324999 wait complete. 2024-04-25T14:04:29.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.342+0000 7fd7a17f3640 1 Processor -- start 2024-04-25T14:04:29.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.342+0000 7fd7a17f3640 1 -- start start 2024-04-25T14:04:29.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd79c06d800 0x7fd79c1e9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79c160150 0x7fd79c15a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c06c8f0 con 0x7fd79c0b4c10 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c06c610 con 0x7fd79c06d800 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd7a17f3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd79c06dc10 con 0x7fd79c160150 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59844/0 (socket says 172.21.15.150:59844) 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 -- 172.21.15.150:0/3074526407 learned_addr learned my addr 172.21.15.150:0/3074526407 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd79b7fe640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79c160150 0x7fd79c15a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd79affd640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd79c06d800 0x7fd79c1e9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 -- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79c160150 msgr2=0x7fd79c15a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79c160150 0x7fd79c15a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 -- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd79c06d800 msgr2=0x7fd79c1e9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd79c06d800 0x7fd79c1e9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.343+0000 7fd793fff640 1 -- 172.21.15.150:0/3074526407 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c1ea520 con 0x7fd79c0b4c10 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd793fff640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd784000c00 tx=0x7fd7840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:29.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784037040 con 0x7fd79c0b4c10 2024-04-25T14:04:29.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd784042070 con 0x7fd79c0b4c10 2024-04-25T14:04:29.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd784030a50 con 0x7fd79c0b4c10 2024-04-25T14:04:29.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c1ea7f0 con 0x7fd79c0b4c10 2024-04-25T14:04:29.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.344+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c1eadb0 con 0x7fd79c0b4c10 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.345+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c076a40 con 0x7fd79c0b4c10 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.346+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd784030bf0 con 0x7fd79c0b4c10 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.346+0000 7fd7937fe640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 0x7fd77c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.346+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(54..54 src has 1..54) ==== 6001+0+0 (secure 0 0 0) 0x7fd7840b5e50 con 0x7fd79c0b4c10 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.346+0000 7fd79affd640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 0x7fd77c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:29.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.347+0000 7fd79affd640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 0x7fd77c0769c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd78c000f60 tx=0x7fd78c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:29.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.351+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd784080540 con 0x7fd79c0b4c10 2024-04-25T14:04:29.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:29.563+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fd79c159410 con 0x7fd79c0b4c10 2024-04-25T14:04:30.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.194+0000 7fd7937fe640 1 -- 172.21.15.150:0/3074526407 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v28) ==== 172+0+0 (secure 0 0 0) 0x7fd78407fe40 con 0x7fd79c0b4c10 2024-04-25T14:04:30.193 INFO:teuthology.orchestra.run.smithi150.stderr:new fs with metadata pool 7 and data pool 8 2024-04-25T14:04:30.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.197+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 msgr2=0x7fd77c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.197+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 0x7fd77c0769c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd78c000f60 tx=0x7fd78c009040 comp rx=0 tx=0).stop 2024-04-25T14:04:30.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.197+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 msgr2=0x7fd79c14f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.197+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd784000c00 tx=0x7fd7840046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 shutdown_connections 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd77c0744b0 0x7fd77c0769c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd79c160150 0x7fd79c15a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd79c0b4c10 0x7fd79c14f6f0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 --2- 172.21.15.150:0/3074526407 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd79c06d800 0x7fd79c1e9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 >> 172.21.15.150:0/3074526407 conn(0x7fd79c0b7690 msgr2=0x7fd79c0b91b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 shutdown_connections 2024-04-25T14:04:30.198 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.198+0000 7fd7a17f3640 1 -- 172.21.15.150:0/3074526407 wait complete. 2024-04-25T14:04:30.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.336+0000 7f094cd14640 1 Processor -- start 2024-04-25T14:04:30.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.336+0000 7f094cd14640 1 -- start start 2024-04-25T14:04:30.337 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 0x7f0948160220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09480d2e90 0x7f09480cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480b53a0 0x7f09480b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f09480b7db0 con 0x7f09480d2e90 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0948164690 con 0x7f09480b2b10 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094cd14640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09480b8350 con 0x7f09480b53a0 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09480d2e90 0x7f09480cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f09480d2e90 0x7f09480cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59854/0 (socket says 172.21.15.150:59854) 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 -- 172.21.15.150:0/2324194687 learned_addr learned my addr 172.21.15.150:0/2324194687 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0946d9d640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480b53a0 0x7f09480b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094659c640 1 --2- 172.21.15.150:0/2324194687 >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 0x7f0948160220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 -- 172.21.15.150:0/2324194687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480b53a0 msgr2=0x7f09480b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480b53a0 0x7f09480b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 -- 172.21.15.150:0/2324194687 >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 msgr2=0x7f0948160220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 --2- 172.21.15.150:0/2324194687 >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 0x7f0948160220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f0945d9b640 1 -- 172.21.15.150:0/2324194687 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09480b5cf0 con 0x7f09480d2e90 2024-04-25T14:04:30.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.338+0000 7f094659c640 1 --2- 172.21.15.150:0/2324194687 >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 0x7f0948160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.339+0000 7f0945d9b640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480d2e90 0x7f09480cdf70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f09480b6140 tx=0x7f093c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90019a1ef256c938 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.339+0000 7f094559a640 1 -- 172.21.15.150:0/2324194687 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c037070 con 0x7f09480d2e90 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.339+0000 7f094559a640 1 -- 172.21.15.150:0/2324194687 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f093c02ae00 con 0x7f09480d2e90 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.339+0000 7f094559a640 1 -- 172.21.15.150:0/2324194687 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093c032660 con 0x7f09480d2e90 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 -- 172.21.15.150:0/2324194687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480d2e90 msgr2=0x7f09480cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480d2e90 0x7f09480cdf70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f09480b6140 tx=0x7f093c02a0b0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 -- 172.21.15.150:0/2324194687 shutdown_connections 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480b53a0 0x7f09480b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 --2- 172.21.15.150:0/2324194687 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480d2e90 0x7f09480cdf70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 --2- 172.21.15.150:0/2324194687 >> v2:172.21.15.187:3300/0 conn(0x7f09480b2b10 0x7f0948160220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 -- 172.21.15.150:0/2324194687 >> 172.21.15.150:0/2324194687 conn(0x7f09480d03a0 msgr2=0x7f09480d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 -- 172.21.15.150:0/2324194687 shutdown_connections 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.340+0000 7f094cd14640 1 -- 172.21.15.150:0/2324194687 wait complete. 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.342+0000 7f094cd14640 1 Processor -- start 2024-04-25T14:04:30.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.342+0000 7f094cd14640 1 -- start start 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480d2e90 0x7f09480c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0948155530 0x7f094815c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f09480b7fe0 con 0x7f09480b53a0 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f09481648e0 con 0x7f0948155530 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094cd14640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f09480b9720 con 0x7f09480d2e90 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59864/0 (socket says 172.21.15.150:59864) 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f094659c640 1 -- 172.21.15.150:0/3711604217 learned_addr learned my addr 172.21.15.150:0/3711604217 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f0946d9d640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0948155530 0x7f094815c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.343+0000 7f0945d9b640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480d2e90 0x7f09480c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 -- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480d2e90 msgr2=0x7f09480c6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480d2e90 0x7f09480c6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 -- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0948155530 msgr2=0x7f094815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0948155530 0x7f094815c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 -- 172.21.15.150:0/3711604217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09480c6b40 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094659c640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f093800d7d0 tx=0x7f093800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0938004030 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09380041d0 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0948155ca0 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f093800b660 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.344+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0948161760 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.345+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0948161270 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.346+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f093800b800 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.346+0000 7f0936ffd640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 0x7f09280769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.346+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0938095560 con 0x7f09480b53a0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.346+0000 7f0945d9b640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 0x7f09280769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.347+0000 7f0945d9b640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 0x7f09280769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f093c034000 tx=0x7f093c02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.351+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f09480c6b40 con 0x7f09480b53a0 2024-04-25T14:04:30.527 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.528+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0948076a40 con 0x7f09480b53a0 2024-04-25T14:04:30.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.530+0000 7f0936ffd640 1 -- 172.21.15.150:0/3711604217 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16955 (secure 0 0 0) 0x7f093805f490 con 0x7f09480b53a0 2024-04-25T14:04:30.529 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:30.529 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":55,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:29.564618+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:30.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 msgr2=0x7f09280769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 0x7f09280769f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f093c034000 tx=0x7f093c02e000 comp rx=0 tx=0).stop 2024-04-25T14:04:30.531 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 msgr2=0x7f09481564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f093800d7d0 tx=0x7f093800dc90 comp rx=0 tx=0).stop 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 shutdown_connections 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f09280744e0 0x7f09280769f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0948155530 0x7f094815c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f09480d2e90 0x7f09480c6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 --2- 172.21.15.150:0/3711604217 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f09480b53a0 0x7f09481564c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 >> 172.21.15.150:0/3711604217 conn(0x7f09480d03a0 msgr2=0x7f094815e2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 shutdown_connections 2024-04-25T14:04:30.532 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.532+0000 7f094cd14640 1 -- 172.21.15.150:0/3711604217 wait complete. 2024-04-25T14:04:30.543 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:30.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.651+0000 7f9403db7640 1 Processor -- start 2024-04-25T14:04:30.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.652+0000 7f9403db7640 1 -- start start 2024-04-25T14:04:30.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93fc160210 0x7f93fc0d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc0b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f93fc0b58e0 0x7f93fc163be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f93fc0b5cf0 con 0x7f93fc160210 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f93fc0d20a0 con 0x7f93fc0b58e0 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9403db7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93fc0d2380 con 0x7f93fc0b2ba0 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9401b53640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93fc160210 0x7f93fc0d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9401b53640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f93fc160210 0x7f93fc0d2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59868/0 (socket says 172.21.15.150:59868) 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9401b53640 1 -- 172.21.15.150:0/2889359986 learned_addr learned my addr 172.21.15.150:0/2889359986 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9402354640 1 --2- 172.21.15.150:0/2889359986 >> v2:172.21.15.187:3300/0 conn(0x7f93fc0b58e0 0x7f93fc163be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.653+0000 7f9401352640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc0b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 -- 172.21.15.150:0/2889359986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 msgr2=0x7f93fc0b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc0b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 -- 172.21.15.150:0/2889359986 >> v2:172.21.15.187:3300/0 conn(0x7f93fc0b58e0 msgr2=0x7f93fc163be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 --2- 172.21.15.150:0/2889359986 >> v2:172.21.15.187:3300/0 conn(0x7f93fc0b58e0 0x7f93fc163be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 -- 172.21.15.150:0/2889359986 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93fc0cfc30 con 0x7f93fc160210 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9401b53640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc160210 0x7f93fc0d2bd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0b64f0 tx=0x7f93f802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960dd661f4b683ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9400b51640 1 -- 172.21.15.150:0/2889359986 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f8036070 con 0x7f93fc160210 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9400b51640 1 -- 172.21.15.150:0/2889359986 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93f8031600 con 0x7f93fc160210 2024-04-25T14:04:30.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.654+0000 7f9400b51640 1 -- 172.21.15.150:0/2889359986 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93f80319d0 con 0x7f93fc160210 2024-04-25T14:04:30.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 -- 172.21.15.150:0/2889359986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc160210 msgr2=0x7f93fc0d2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.656 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc160210 0x7f93fc0d2bd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0b64f0 tx=0x7f93f802a6a0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 -- 172.21.15.150:0/2889359986 shutdown_connections 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 --2- 172.21.15.150:0/2889359986 >> v2:172.21.15.187:3300/0 conn(0x7f93fc0b58e0 0x7f93fc163be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc0b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 --2- 172.21.15.150:0/2889359986 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc160210 0x7f93fc0d2bd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 -- 172.21.15.150:0/2889359986 >> 172.21.15.150:0/2889359986 conn(0x7f93fc0d03a0 msgr2=0x7f93fc0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 -- 172.21.15.150:0/2889359986 shutdown_connections 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.655+0000 7f9403db7640 1 -- 172.21.15.150:0/2889359986 wait complete. 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.657+0000 7f9403db7640 1 Processor -- start 2024-04-25T14:04:30.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.657+0000 7f9403db7640 1 -- start start 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc161a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 0x7f93fc1f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc164120 con 0x7f93fc0b58e0 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f93fc0b2fb0 con 0x7f93fc160210 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9403db7640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f93fc1642a0 con 0x7f93fc0b2ba0 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9401352640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9401352640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59872/0 (socket says 172.21.15.150:59872) 2024-04-25T14:04:30.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.658+0000 7f9401352640 1 -- 172.21.15.150:0/1867729750 learned_addr learned my addr 172.21.15.150:0/1867729750 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9402354640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 0x7f93fc1f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401b53640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc161a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 -- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 msgr2=0x7f93fc161a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc161a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 -- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 msgr2=0x7f93fc1f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 0x7f93fc1f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 -- 172.21.15.150:0/1867729750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93fc161fc0 con 0x7f93fc0b58e0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9402354640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 0x7f93fc1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9401352640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00e870 tx=0x7f93ec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec017070 con 0x7f93fc0b58e0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93ec00c940 con 0x7f93fc0b58e0 2024-04-25T14:04:30.661 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93fc1633a0 con 0x7f93fc0b58e0 2024-04-25T14:04:30.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ec00cd30 con 0x7f93fc0b58e0 2024-04-25T14:04:30.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.659+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93fc161280 con 0x7f93fc0b58e0 2024-04-25T14:04:30.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.660+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93fc163110 con 0x7f93fc0b58e0 2024-04-25T14:04:30.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.661+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f93ec01a070 con 0x7f93fc0b58e0 2024-04-25T14:04:30.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.661+0000 7f93f27fc640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 0x7f93e00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.662+0000 7f9401b53640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 0x7f93e00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.662+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f93ec095240 con 0x7f93fc0b58e0 2024-04-25T14:04:30.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.662+0000 7f9401b53640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 0x7f93e00769c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93f8033000 tx=0x7f93f8009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.665 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.666+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f93fc161fc0 con 0x7f93fc0b58e0 2024-04-25T14:04:30.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.843+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93fc076a40 con 0x7f93fc0b58e0 2024-04-25T14:04:30.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.844+0000 7f93f27fc640 1 -- 172.21.15.150:0/1867729750 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) ==== 74+0+16955 (secure 0 0 0) 0x7f93ec05f1f0 con 0x7f93fc0b58e0 2024-04-25T14:04:30.844 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:30.844 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":55,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:29.564618+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:30.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 msgr2=0x7f93e00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 0x7f93e00769c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f93f8033000 tx=0x7f93f8009000 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 msgr2=0x7f93fc151be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f93ec00e870 tx=0x7f93ec00ed30 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 shutdown_connections 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f93e00744b0 0x7f93e00769c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f93fc160210 0x7f93fc1f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f93fc0b58e0 0x7f93fc151be0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 --2- 172.21.15.150:0/1867729750 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f93fc0b2ba0 0x7f93fc161a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 >> 172.21.15.150:0/1867729750 conn(0x7f93fc0d03a0 msgr2=0x7f93fc15c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 shutdown_connections 2024-04-25T14:04:30.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.846+0000 7f9403db7640 1 -- 172.21.15.150:0/1867729750 wait complete. 2024-04-25T14:04:30.857 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T14:04:30.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.968+0000 7f0b47df8640 1 Processor -- start 2024-04-25T14:04:30.967 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.968+0000 7f0b47df8640 1 -- start start 2024-04-25T14:04:30.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b47df8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f0b4015e4e0 0x7f0b400b2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b47df8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b4009b4a0 0x7f0b4009b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b47df8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b4009bdf0 con 0x7f0b4006f640 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b47df8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4015e9c0 con 0x7f0b4009b4a0 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b47df8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f0b4015eca0 con 0x7f0b4015e4e0 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b4009b4a0 0x7f0b4009b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f0b4009b4a0 0x7f0b4009b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59886/0 (socket says 172.21.15.150:59886) 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 -- 172.21.15.150:0/243391374 learned_addr learned my addr 172.21.15.150:0/243391374 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b45b94640 1 --2- 172.21.15.150:0/243391374 >> v2:172.21.15.187:3300/0 conn(0x7f0b4015e4e0 0x7f0b400b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 -- 172.21.15.150:0/243391374 >> v2:172.21.15.187:3300/0 conn(0x7f0b4015e4e0 msgr2=0x7f0b400b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 --2- 172.21.15.150:0/243391374 >> v2:172.21.15.187:3300/0 conn(0x7f0b4015e4e0 0x7f0b400b2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 -- 172.21.15.150:0/243391374 >> v1:172.21.15.187:6789/0 conn(0x7f0b4006f640 legacy=0x7f0b4006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:30.970 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.970+0000 7f0b46395640 1 -- 172.21.15.150:0/243391374 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4015ef80 con 0x7f0b4009b4a0 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.971+0000 7f0b46395640 1 --2- 172.21.15.150:0/243391374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b4009b8b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0b40147c80 tx=0x7f0b3802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783dfea37b3b0958 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.971+0000 7f0b44b92640 1 -- 172.21.15.150:0/243391374 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b38036070 con 0x7f0b4009b4a0 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.971+0000 7f0b44b92640 1 -- 172.21.15.150:0/243391374 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b38031600 con 0x7f0b4009b4a0 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.971+0000 7f0b44b92640 1 -- 172.21.15.150:0/243391374 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b380319d0 con 0x7f0b4009b4a0 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 -- 172.21.15.150:0/243391374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 msgr2=0x7f0b4009b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 --2- 172.21.15.150:0/243391374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b4009b8b0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0b40147c80 tx=0x7f0b3802a6f0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 -- 172.21.15.150:0/243391374 shutdown_connections 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 --2- 172.21.15.150:0/243391374 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b4009b8b0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 --2- 172.21.15.150:0/243391374 >> v2:172.21.15.187:3300/0 conn(0x7f0b4015e4e0 0x7f0b400b2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 -- 172.21.15.150:0/243391374 >> 172.21.15.150:0/243391374 conn(0x7f0b400b7680 msgr2=0x7f0b400b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:30.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 -- 172.21.15.150:0/243391374 shutdown_connections 2024-04-25T14:04:30.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.972+0000 7f0b47df8640 1 -- 172.21.15.150:0/243391374 wait complete. 2024-04-25T14:04:30.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.974+0000 7f0b47df8640 1 Processor -- start 2024-04-25T14:04:30.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.974+0000 7f0b47df8640 1 -- start start 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b4006f640 0x7f0b401444c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b4015e4e0 0x7f0b4015a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f0b400b2890 con 0x7f0b4009b4a0 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f0b4006d660 con 0x7f0b4006f640 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b47df8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f0b400b2a10 con 0x7f0b4015e4e0 2024-04-25T14:04:30.975 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59890/0 (socket says 172.21.15.150:59890) 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 -- 172.21.15.150:0/2130920809 learned_addr learned my addr 172.21.15.150:0/2130920809 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45b94640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b4006f640 0x7f0b401444c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b46395640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b4015e4e0 0x7f0b4015a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 -- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b4015e4e0 msgr2=0x7f0b4015a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b4015e4e0 0x7f0b4015a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 -- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b4006f640 msgr2=0x7f0b401444c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b4006f640 0x7f0b401444c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.975+0000 7f0b45393640 1 -- 172.21.15.150:0/2130920809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b4015a840 con 0x7f0b4009b4a0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b45393640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000b420 tx=0x7f0b3000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30011830 con 0x7f0b4009b4a0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b30016070 con 0x7f0b4009b4a0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b4015ab20 con 0x7f0b4009b4a0 2024-04-25T14:04:30.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30011c10 con 0x7f0b4009b4a0 2024-04-25T14:04:30.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.976+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b4015b0e0 con 0x7f0b4009b4a0 2024-04-25T14:04:30.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.977+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b4015a5b0 con 0x7f0b4009b4a0 2024-04-25T14:04:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.977+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0b3001b460 con 0x7f0b4009b4a0 2024-04-25T14:04:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.978+0000 7f0b367fc640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.978+0000 7f0b45b94640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.978+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 6101+0+0 (secure 0 0 0) 0x7f0b30094f10 con 0x7f0b4009b4a0 2024-04-25T14:04:30.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.978+0000 7f0b45b94640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 0x7f0b240769c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c006fd0 tx=0x7f0b3c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:30.981 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:30.983+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b30020020 con 0x7f0b4009b4a0 2024-04-25T14:04:31.193 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.194+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f0b40076a40 con 0x7f0b4009b4a0 2024-04-25T14:04:31.210 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.211+0000 7f0b367fc640 1 -- 172.21.15.150:0/2130920809 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) ==== 118+0+0 (secure 0 0 0) 0x7f0b3005f510 con 0x7f0b4009b4a0 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 msgr2=0x7f0b240769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 0x7f0b240769c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c006fd0 tx=0x7f0b3c008040 comp rx=0 tx=0).stop 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 msgr2=0x7f0b40159b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0b3000b420 tx=0x7f0b3000b8e0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 shutdown_connections 2024-04-25T14:04:31.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f0b240744b0 0x7f0b240769c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f0b4015e4e0 0x7f0b4015a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f0b4009b4a0 0x7f0b40159b30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 --2- 172.21.15.150:0/2130920809 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f0b4006f640 0x7f0b401444c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 >> 172.21.15.150:0/2130920809 conn(0x7f0b400b7680 msgr2=0x7f0b40070440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.213+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 shutdown_connections 2024-04-25T14:04:31.214 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.214+0000 7f0b47df8640 1 -- 172.21.15.150:0/2130920809 wait complete. 2024-04-25T14:04:31.224 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-25T14:04:31.224 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-25T14:04:31.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.335+0000 7fefff627640 1 Processor -- start 2024-04-25T14:04:31.335 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.336+0000 7fefff627640 1 -- start start 2024-04-25T14:04:31.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7fefff627640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7fefff627640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7fefff627640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feff80737f0 con 0x7feff80d18b0 2024-04-25T14:04:31.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7fefff627640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7feff80d2e90 con 0x7feff80b6800 2024-04-25T14:04:31.336 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7fefff627640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feff80d0080 con 0x7feff8072ea0 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7feffcbc2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7feffd3c3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7feffcbc2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:54298/0 (socket says 172.21.15.150:54298) 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7feffd3c3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:47050/0 (socket says 172.21.15.150:47050) 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.337+0000 7feffcbc2640 1 -- 172.21.15.150:0/1225058160 learned_addr learned my addr 172.21.15.150:0/1225058160 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7fefeffff640 1 -- 172.21.15.150:0/1225058160 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1588522862 0 0) 0x7feff0001110 con 0x7feff80d18b0 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7fefeffff640 1 -- 172.21.15.150:0/1225058160 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefe4003520 con 0x7feff80d18b0 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffcbc2640 1 -- 172.21.15.150:0/1225058160 >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 msgr2=0x7feff81603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:31.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffcbc2640 1 --2- 172.21.15.150:0/1225058160 >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffcbc2640 1 -- 172.21.15.150:0/1225058160 >> v1:172.21.15.187:6789/0 conn(0x7feff80d18b0 legacy=0x7feff80d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffcbc2640 1 -- 172.21.15.150:0/1225058160 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff8164a50 con 0x7feff8072ea0 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffd3c3640 1 --2- 172.21.15.150:0/1225058160 >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.338+0000 7feffcbc2640 1 --2- 172.21.15.150:0/1225058160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009870 tx=0x7fefe802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dc1bf921f0b7327 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefeffff640 1 -- 172.21.15.150:0/1225058160 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8037070 con 0x7feff8072ea0 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefeffff640 1 -- 172.21.15.150:0/1225058160 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe8032440 con 0x7feff8072ea0 2024-04-25T14:04:31.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefeffff640 1 -- 172.21.15.150:0/1225058160 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe8032810 con 0x7feff8072ea0 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 -- 172.21.15.150:0/1225058160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 msgr2=0x7feff80732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 --2- 172.21.15.150:0/1225058160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009870 tx=0x7fefe802a1c0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 -- 172.21.15.150:0/1225058160 shutdown_connections 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 --2- 172.21.15.150:0/1225058160 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff8072ea0 0x7feff80732b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 --2- 172.21.15.150:0/1225058160 >> v2:172.21.15.187:3300/0 conn(0x7feff80b6800 0x7feff81603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 -- 172.21.15.150:0/1225058160 >> 172.21.15.150:0/1225058160 conn(0x7feff80d39b0 msgr2=0x7feff80d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:31.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 -- 172.21.15.150:0/1225058160 shutdown_connections 2024-04-25T14:04:31.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.339+0000 7fefff627640 1 -- 172.21.15.150:0/1225058160 wait complete. 2024-04-25T14:04:31.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.341+0000 7fefff627640 1 Processor -- start 2024-04-25T14:04:31.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.341+0000 7fefff627640 1 -- start start 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 0x7feff8162970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 0x7feff8162eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7feff80cfd10 con 0x7feff8072ea0 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7feff80cfa30 con 0x7feff80b6800 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.342+0000 7fefff627640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7feff80b98e0 con 0x7feff80d18b0 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59906/0 (socket says 172.21.15.150:59906) 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 -- 172.21.15.150:0/297214631 learned_addr learned my addr 172.21.15.150:0/297214631 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffcbc2640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 0x7feff8162970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffdbc4640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 0x7feff8162eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 -- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 msgr2=0x7feff8162eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 0x7feff8162eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 -- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 msgr2=0x7feff8162970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 0x7feff8162970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 -- 172.21.15.150:0/297214631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feff8160cd0 con 0x7feff8072ea0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffdbc4640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 0x7feff8162eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffcbc2640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 0x7feff8162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7feffd3c3640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7feff4009e60 tx=0x7feff400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4018020 con 0x7feff8072ea0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feff4014680 con 0x7feff8072ea0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feff8153fb0 con 0x7feff8072ea0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.343+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4014a50 con 0x7feff8072ea0 2024-04-25T14:04:31.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.344+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feff8154570 con 0x7feff8072ea0 2024-04-25T14:04:31.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.344+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feff815bca0 con 0x7feff8072ea0 2024-04-25T14:04:31.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.345+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7feff4005d10 con 0x7feff8072ea0 2024-04-25T14:04:31.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.345+0000 7fefedffb640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 0x7fefdc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:31.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.346+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7feff4096140 con 0x7feff8072ea0 2024-04-25T14:04:31.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.346+0000 7feffcbc2640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 0x7fefdc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:31.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.346+0000 7feffcbc2640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 0x7fefdc0769f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fefe8034000 tx=0x7fefe802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:31.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.350+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7feff815bca0 con 0x7feff8072ea0 2024-04-25T14:04:31.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:31.564+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7feff80697d0 con 0x7feff8072ea0 2024-04-25T14:04:32.212 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.213+0000 7fefedffb640 1 -- 172.21.15.150:0/297214631 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v30) ==== 122+0+0 (secure 0 0 0) 0x7feff80697d0 con 0x7feff8072ea0 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 msgr2=0x7fefdc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 0x7fefdc0769f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fefe8034000 tx=0x7fefe802e000 comp rx=0 tx=0).stop 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 msgr2=0x7feff81508f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7feff4009e60 tx=0x7feff400c490 comp rx=0 tx=0).stop 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 shutdown_connections 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fefdc0744e0 0x7fefdc0769f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7feff80d18b0 0x7feff8162eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.215 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7feff80b6800 0x7feff8162970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 --2- 172.21.15.150:0/297214631 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7feff8072ea0 0x7feff81508f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.215+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 >> 172.21.15.150:0/297214631 conn(0x7feff80d39b0 msgr2=0x7feff80d1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:32.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.216+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 shutdown_connections 2024-04-25T14:04:32.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.216+0000 7fefff627640 1 -- 172.21.15.150:0/297214631 wait complete. 2024-04-25T14:04:32.228 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-25T14:04:32.228 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:32.339 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.340+0000 7f045ba67640 1 Processor -- start 2024-04-25T14:04:32.340 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.341+0000 7f045ba67640 1 -- start start 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045ba67640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f045415df00 0x7f045406f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045ba67640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f045406f780 0x7f04540b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045ba67640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04540b6140 con 0x7f04540b5d30 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045ba67640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f045415e310 con 0x7f045406f780 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045ba67640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f04540b5360 con 0x7f045415df00 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f0459002640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f045406f780 0x7f04540b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045a004640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f04540b5d30 0x7f04540ae7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41180/0 (socket says 172.21.15.150:41180) 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f045a004640 1 -- 172.21.15.150:0/1242229590 learned_addr learned my addr 172.21.15.150:0/1242229590 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.342+0000 7f0459803640 1 --2- 172.21.15.150:0/1242229590 >> v2:172.21.15.187:3300/0 conn(0x7f045415df00 0x7f045406f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0459002640 1 -- 172.21.15.150:0/1242229590 >> v2:172.21.15.187:3300/0 conn(0x7f045415df00 msgr2=0x7f045406f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0459002640 1 --2- 172.21.15.150:0/1242229590 >> v2:172.21.15.187:3300/0 conn(0x7f045415df00 0x7f045406f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0459002640 1 -- 172.21.15.150:0/1242229590 >> v1:172.21.15.150:6789/0 conn(0x7f04540b5d30 legacy=0x7f04540ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0459002640 1 -- 172.21.15.150:0/1242229590 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04540b4750 con 0x7f045406f780 2024-04-25T14:04:32.343 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0458801640 1 -- 172.21.15.150:0/1242229590 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2421594581 0 0) 0x7f04540b6140 con 0x7f04540b5d30 2024-04-25T14:04:32.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.343+0000 7f0459002640 1 --2- 172.21.15.150:0/1242229590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f045406f780 0x7f04540b57f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f045406c680 tx=0x7f044402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d230031555ba93 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f0458801640 1 -- 172.21.15.150:0/1242229590 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444037070 con 0x7f045406f780 2024-04-25T14:04:32.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f0458801640 1 -- 172.21.15.150:0/1242229590 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444032440 con 0x7f045406f780 2024-04-25T14:04:32.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f0458801640 1 -- 172.21.15.150:0/1242229590 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044402ae60 con 0x7f045406f780 2024-04-25T14:04:32.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 -- 172.21.15.150:0/1242229590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f045406f780 msgr2=0x7f04540b57f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 --2- 172.21.15.150:0/1242229590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f045406f780 0x7f04540b57f0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f045406c680 tx=0x7f044402a110 comp rx=0 tx=0).stop 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 -- 172.21.15.150:0/1242229590 shutdown_connections 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 --2- 172.21.15.150:0/1242229590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f045406f780 0x7f04540b57f0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 --2- 172.21.15.150:0/1242229590 >> v2:172.21.15.187:3300/0 conn(0x7f045415df00 0x7f045406f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.344+0000 7f045ba67640 1 -- 172.21.15.150:0/1242229590 >> 172.21.15.150:0/1242229590 conn(0x7f04540d1ae0 msgr2=0x7f04540d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.345+0000 7f045ba67640 1 -- 172.21.15.150:0/1242229590 shutdown_connections 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.345+0000 7f045ba67640 1 -- 172.21.15.150:0/1242229590 wait complete. 2024-04-25T14:04:32.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.346+0000 7f045ba67640 1 Processor -- start 2024-04-25T14:04:32.346 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.346+0000 7f045ba67640 1 -- start start 2024-04-25T14:04:32.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f045406f780 0x7f045413cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 0x7f04540cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f04540b4ff0 con 0x7f04540b5d30 2024-04-25T14:04:32.348 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f04540b4d10 con 0x7f045406f780 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045ba67640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f04540aeef0 con 0x7f045415df00 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59918/0 (socket says 172.21.15.150:59918) 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 -- 172.21.15.150:0/3367854064 learned_addr learned my addr 172.21.15.150:0/3367854064 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459803640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f045406f780 0x7f045413cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045a004640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 0x7f04540cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 -- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 msgr2=0x7f04540cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 0x7f04540cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 -- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f045406f780 msgr2=0x7f045413cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f045406f780 0x7f045413cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 -- 172.21.15.150:0/3367854064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045413d330 con 0x7f04540b5d30 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f045a004640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 0x7f04540cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f0459002640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0444009890 tx=0x7f04440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.348+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444037040 con 0x7f04540b5d30 2024-04-25T14:04:32.349 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.349+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444042070 con 0x7f04540b5d30 2024-04-25T14:04:32.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.349+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444030920 con 0x7f04540b5d30 2024-04-25T14:04:32.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.349+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f045413d610 con 0x7f04540b5d30 2024-04-25T14:04:32.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.349+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f045413af20 con 0x7f04540b5d30 2024-04-25T14:04:32.350 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.350+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0454076a40 con 0x7f04540b5d30 2024-04-25T14:04:32.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.350+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0444030ac0 con 0x7f04540b5d30 2024-04-25T14:04:32.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.351+0000 7f044a7fc640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 0x7f04380769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.351+0000 7f0459803640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 0x7f04380769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.351+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f04440b5d00 con 0x7f04540b5d30 2024-04-25T14:04:32.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.351+0000 7f0459803640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 0x7f04380769f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0450009a30 tx=0x7f0450008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.356+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f044407fcb0 con 0x7f04540b5d30 2024-04-25T14:04:32.566 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.567+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f045413d060 con 0x7f04540b5d30 2024-04-25T14:04:32.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.568+0000 7f044a7fc640 1 -- 172.21.15.150:0/3367854064 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5187 (secure 0 0 0) 0x7f0444083fb0 con 0x7f04540b5d30 2024-04-25T14:04:32.567 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:32.567 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/468558883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":468558883},{"type":"v1","addr":"172.21.15.187:6834","nonce":468558883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3011235888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3011235888},{"type":"v1","addr":"172.21.15.150:6835","nonce":3011235888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5823,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1873279280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1873279280},{"type":"v1","addr":"172.21.15.150:6837","nonce":1873279280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:31.564985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5832},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5832":{"gid":5832,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/133937444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":133937444},{"type":"v1","addr":"172.21.15.187:6835","nonce":133937444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5832,"qdb_cluster":[5832]},"id":3}]} 2024-04-25T14:04:32.568 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 30 2024-04-25T14:04:32.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 msgr2=0x7f04380769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 0x7f04380769f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f0450009a30 tx=0x7f0450008040 comp rx=0 tx=0).stop 2024-04-25T14:04:32.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 msgr2=0x7f04541f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0444009890 tx=0x7f04440046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 shutdown_connections 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f04380744e0 0x7f04380769f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f045415df00 0x7f04540cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f04540b5d30 0x7f04541f6060 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 --2- 172.21.15.150:0/3367854064 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f045406f780 0x7f045413cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 >> 172.21.15.150:0/3367854064 conn(0x7f04540d1ae0 msgr2=0x7f0454070ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 shutdown_connections 2024-04-25T14:04:32.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.570+0000 7f045ba67640 1 -- 172.21.15.150:0/3367854064 wait complete. 2024-04-25T14:04:32.581 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:04:32.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.690+0000 7fe462f7d640 1 Processor -- start 2024-04-25T14:04:32.689 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.690+0000 7fe462f7d640 1 -- start start 2024-04-25T14:04:32.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.691+0000 7fe462f7d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe45c15e170 0x7fe45c0ce180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.691+0000 7fe462f7d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 0x7fe45c0b53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.691+0000 7fe462f7d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe45c0b5cf0 con 0x7fe45c0b58e0 2024-04-25T14:04:32.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.691+0000 7fe462f7d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe45c0b6a80 con 0x7fe45c15e170 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.691+0000 7fe462f7d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe45c0ceb90 con 0x7fe45c0b2850 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe460d19640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe45c15e170 0x7fe45c0ce180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe45bfff640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 0x7fe45c0b53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe46151a640 1 --1- >> v1:172.21.15.187:6789/0 conn(0x7fe45c0b58e0 0x7fe45c070120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.187:6789/0 says I am v1:172.21.15.150:56296/0 (socket says 172.21.15.150:56296) 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe46151a640 1 -- 172.21.15.150:0/1585528871 learned_addr learned my addr 172.21.15.150:0/1585528871 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe45b7fe640 1 -- 172.21.15.150:0/1585528871 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1443542376 0 0) 0x7fe454001490 con 0x7fe45c0b58e0 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe45b7fe640 1 -- 172.21.15.150:0/1585528871 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe44c003520 con 0x7fe45c0b58e0 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe460d19640 1 -- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 msgr2=0x7fe45c0b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe460d19640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 0x7fe45c0b53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe460d19640 1 -- 172.21.15.150:0/1585528871 >> v1:172.21.15.187:6789/0 conn(0x7fe45c0b58e0 legacy=0x7fe45c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe460d19640 1 -- 172.21.15.150:0/1585528871 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c162490 con 0x7fe45c15e170 2024-04-25T14:04:32.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.692+0000 7fe45bfff640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 0x7fe45c0b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:04:32.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.693+0000 7fe460d19640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c15e170 0x7fe45c0ce180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7236469249cf1b3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.693+0000 7fe45b7fe640 1 -- 172.21.15.150:0/1585528871 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe450037070 con 0x7fe45c15e170 2024-04-25T14:04:32.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.693+0000 7fe45b7fe640 1 -- 172.21.15.150:0/1585528871 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe450032440 con 0x7fe45c15e170 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.693+0000 7fe45b7fe640 1 -- 172.21.15.150:0/1585528871 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4500327f0 con 0x7fe45c15e170 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 -- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c15e170 msgr2=0x7fe45c0ce180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c15e170 0x7fe45c0ce180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe4500097a0 tx=0x7fe45002a210 comp rx=0 tx=0).stop 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 -- 172.21.15.150:0/1585528871 shutdown_connections 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c0b2850 0x7fe45c0b53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 --2- 172.21.15.150:0/1585528871 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c15e170 0x7fe45c0ce180 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 -- 172.21.15.150:0/1585528871 >> 172.21.15.150:0/1585528871 conn(0x7fe45c0d03a0 msgr2=0x7fe45c0d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 -- 172.21.15.150:0/1585528871 shutdown_connections 2024-04-25T14:04:32.694 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.694+0000 7fe462f7d640 1 -- 172.21.15.150:0/1585528871 wait complete. 2024-04-25T14:04:32.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.696+0000 7fe462f7d640 1 Processor -- start 2024-04-25T14:04:32.695 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.696+0000 7fe462f7d640 1 -- start start 2024-04-25T14:04:32.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c0b2850 0x7fe45c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 0x7fe45c1f7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c162750 con 0x7fe45c0b58e0 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe45c0b3080 con 0x7fe45c0b2850 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe462f7d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe45c0d3010 con 0x7fe45c15e170 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe45bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe45bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59928/0 (socket says 172.21.15.150:59928) 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe45bfff640 1 -- 172.21.15.150:0/283994026 learned_addr learned my addr 172.21.15.150:0/283994026 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe460d19640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c0b2850 0x7fe45c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.697+0000 7fe46151a640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 0x7fe45c1f7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 -- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 msgr2=0x7fe45c1f7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 0x7fe45c1f7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 -- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c0b2850 msgr2=0x7fe45c0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c0b2850 0x7fe45c0cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 -- 172.21.15.150:0/283994026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45c15feb0 con 0x7fe45c0b58e0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe46151a640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 0x7fe45c1f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe45bfff640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe44800e990 tx=0x7fe44800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448017070 con 0x7fe45c0b58e0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44800c9e0 con 0x7fe45c0b58e0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45c1f8970 con 0x7fe45c0b58e0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44800cdb0 con 0x7fe45c0b58e0 2024-04-25T14:04:32.699 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.698+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45c15f130 con 0x7fe45c0b58e0 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.699+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe45c076a40 con 0x7fe45c0b58e0 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.699+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe44801a070 con 0x7fe45c0b58e0 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.700+0000 7fe4597fa640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 0x7fe4400769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.700+0000 7fe460d19640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 0x7fe4400769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.700+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fe448095070 con 0x7fe45c0b58e0 2024-04-25T14:04:32.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.700+0000 7fe460d19640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 0x7fe4400769f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe450034000 tx=0x7fe45002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:32.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.704+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe44805f120 con 0x7fe45c0b58e0 2024-04-25T14:04:32.880 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.881+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe45c0ce760 con 0x7fe45c0b58e0 2024-04-25T14:04:32.881 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.882+0000 7fe4597fa640 1 -- 172.21.15.150:0/283994026 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) ==== 74+0+16942 (secure 0 0 0) 0x7fe448063420 con 0x7fe45c0b58e0 2024-04-25T14:04:32.881 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:32.881 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":56,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:31.204750+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:04:32.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 msgr2=0x7fe4400769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 0x7fe4400769f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe450034000 tx=0x7fe45002e000 comp rx=0 tx=0).stop 2024-04-25T14:04:32.883 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 msgr2=0x7fe45c1f3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fe44800e990 tx=0x7fe44800ee50 comp rx=0 tx=0).stop 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 shutdown_connections 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4400744e0 0x7fe4400769f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe45c15e170 0x7fe45c1f7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe45c0b58e0 0x7fe45c1f3ed0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 --2- 172.21.15.150:0/283994026 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe45c0b2850 0x7fe45c0cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 >> 172.21.15.150:0/283994026 conn(0x7fe45c0d03a0 msgr2=0x7fe45c0d2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 shutdown_connections 2024-04-25T14:04:32.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:32.884+0000 7fe462f7d640 1 -- 172.21.15.150:0/283994026 wait complete. 2024-04-25T14:04:32.895 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T14:04:32.895 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:04:33.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.003+0000 7fbcde54e640 1 Processor -- start 2024-04-25T14:04:33.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.003+0000 7fbcde54e640 1 -- start start 2024-04-25T14:04:33.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcde54e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcde54e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd8164910 con 0x7fbcd8160240 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcde54e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcd80cfdb0 con 0x7fbcd80b5ab0 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcde54e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbcd80b80b0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcdcaeb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcd7fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbcd8160240 0x7fbcd80b53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41194/0 (socket says 172.21.15.150:41194) 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcdcaeb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59932/0 (socket says 172.21.15.150:59932) 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcd7fff640 1 -- 172.21.15.150:0/3431457433 learned_addr learned my addr 172.21.15.150:0/3431457433 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcdcaeb640 1 -- 172.21.15.150:0/3431457433 >> v1:172.21.15.187:6789/0 conn(0x7fbcd80b5ab0 legacy=0x7fbcd80b8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcdcaeb640 1 -- 172.21.15.150:0/3431457433 >> v1:172.21.15.150:6789/0 conn(0x7fbcd8160240 legacy=0x7fbcd80b53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcdcaeb640 1 -- 172.21.15.150:0/3431457433 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80d0090 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.005+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/3431457433 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3138041397 0 0) 0x7fbcd8164910 con 0x7fbcd8160240 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.006+0000 7fbcdcaeb640 1 --2- 172.21.15.150:0/3431457433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009d90 tx=0x7fbcd002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=225c65b036bf1b6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.006+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/3431457433 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0007740 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.006+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/3431457433 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcd00078e0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.006+0000 7fbcd6ffd640 1 -- 172.21.15.150:0/3431457433 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd0007cb0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 -- 172.21.15.150:0/3431457433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 msgr2=0x7fbcd80d2d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 --2- 172.21.15.150:0/3431457433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fbcd0009d90 tx=0x7fbcd002a6a0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 -- 172.21.15.150:0/3431457433 shutdown_connections 2024-04-25T14:04:33.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 --2- 172.21.15.150:0/3431457433 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd80d2d60 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 -- 172.21.15.150:0/3431457433 >> 172.21.15.150:0/3431457433 conn(0x7fbcd80d03a0 msgr2=0x7fbcd80d07b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 -- 172.21.15.150:0/3431457433 shutdown_connections 2024-04-25T14:04:33.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.007+0000 7fbcde54e640 1 -- 172.21.15.150:0/3431457433 wait complete. 2024-04-25T14:04:33.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.008+0000 7fbcde54e640 1 Processor -- start 2024-04-25T14:04:33.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.009+0000 7fbcde54e640 1 -- start start 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 0x7fbcd80ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcd8160240 0x7fbcd81623c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80b8dd0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbcd80b8c50 con 0x7fbcd80b5ab0 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcde54e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbcd80b8f50 con 0x7fbcd8160240 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59942/0 (socket says 172.21.15.150:59942) 2024-04-25T14:04:33.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 -- 172.21.15.150:0/4233268789 learned_addr learned my addr 172.21.15.150:0/4233268789 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcdcaeb640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcd8160240 0x7fbcd81623c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd77fe640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 0x7fbcd80ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 -- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcd8160240 msgr2=0x7fbcd81623c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcd8160240 0x7fbcd81623c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 -- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 msgr2=0x7fbcd80ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 0x7fbcd80ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd7fff640 1 -- 172.21.15.150:0/4233268789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcd80cb3b0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.010+0000 7fbcd77fe640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 0x7fbcd80ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcd7fff640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800c410 tx=0x7fbcc800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8017070 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcc8005ce0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcd80cb690 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80072f0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.011+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcd81eafa0 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.012+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcd81e9820 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.012+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbcc801a070 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.013+0000 7fbcd4ff9640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 0x7fbcc40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.013+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fbcc8094890 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.013+0000 7fbcd77fe640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 0x7fbcc40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.013+0000 7fbcd77fe640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 0x7fbcc40769c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80b5150 tx=0x7fbccc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.018+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbcd81e9820 con 0x7fbcd80b2ca0 2024-04-25T14:04:33.189 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.190+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbcd80cbbb0 con 0x7fbcc40744b0 2024-04-25T14:04:33.196 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.196+0000 7fbcd4ff9640 1 -- 172.21.15.150:0/4233268789 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251898 (secure 0 0 0) 0x7fbcd80cbbb0 con 0x7fbcc40744b0 2024-04-25T14:04:33.199 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:33.201 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 msgr2=0x7fbcc40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 0x7fbcc40769c0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80b5150 tx=0x7fbccc008040 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 msgr2=0x7fbcd81520c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800c410 tx=0x7fbcc800c8d0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 shutdown_connections 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbcc40744b0 0x7fbcc40769c0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbcd8160240 0x7fbcd81623c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbcd80b5ab0 0x7fbcd80ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 --2- 172.21.15.150:0/4233268789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbcd80b2ca0 0x7fbcd81520c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.199+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 >> 172.21.15.150:0/4233268789 conn(0x7fbcd80d03a0 msgr2=0x7fbcd80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.203 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.200+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 shutdown_connections 2024-04-25T14:04:33.204 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.200+0000 7fbcde54e640 1 -- 172.21.15.150:0/4233268789 wait complete. 2024-04-25T14:04:33.210 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-04-25T14:04:32.619397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222544,"kb_used_data":6368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752176,"statfs":{"total":773094113280,"available":772866228224,"internally_reserved":0,"allocated":6520832,"data_stored":3573572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.431389"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.218963+0000","last_change":"2024-04-25T14:04:30.214547+0000","last_active":"2024-04-25T14:04:30.218963+0000","last_peered":"2024-04-25T14:04:30.218963+0000","last_clean":"2024-04-25T14:04:30.218963+0000","last_became_active":"2024-04-25T14:04:30.214243+0000","last_became_peered":"2024-04-25T14:04:30.214243+0000","last_unstale":"2024-04-25T14:04:30.218963+0000","last_undegraded":"2024-04-25T14:04:30.218963+0000","last_fullsized":"2024-04-25T14:04:30.218963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222180+0000","last_change":"2024-04-25T14:04:30.219227+0000","last_active":"2024-04-25T14:04:31.222180+0000","last_peered":"2024-04-25T14:04:31.222180+0000","last_clean":"2024-04-25T14:04:31.222180+0000","last_became_active":"2024-04-25T14:04:30.218875+0000","last_became_peered":"2024-04-25T14:04:30.218875+0000","last_unstale":"2024-04-25T14:04:31.222180+0000","last_undegraded":"2024-04-25T14:04:31.222180+0000","last_fullsized":"2024-04-25T14:04:31.222180+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231177+0000","last_change":"2024-04-25T14:04:30.226045+0000","last_active":"2024-04-25T14:04:30.231177+0000","last_peered":"2024-04-25T14:04:30.231177+0000","last_clean":"2024-04-25T14:04:30.231177+0000","last_became_active":"2024-04-25T14:04:30.225740+0000","last_became_peered":"2024-04-25T14:04:30.225740+0000","last_unstale":"2024-04-25T14:04:30.231177+0000","last_undegraded":"2024-04-25T14:04:30.231177+0000","last_fullsized":"2024-04-25T14:04:30.231177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222720+0000","last_change":"2024-04-25T14:04:30.214708+0000","last_active":"2024-04-25T14:04:31.222720+0000","last_peered":"2024-04-25T14:04:31.222720+0000","last_clean":"2024-04-25T14:04:31.222720+0000","last_became_active":"2024-04-25T14:04:30.214430+0000","last_became_peered":"2024-04-25T14:04:30.214430+0000","last_unstale":"2024-04-25T14:04:31.222720+0000","last_undegraded":"2024-04-25T14:04:31.222720+0000","last_fullsized":"2024-04-25T14:04:31.222720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221187+0000","last_change":"2024-04-25T14:04:30.215762+0000","last_active":"2024-04-25T14:04:31.221187+0000","last_peered":"2024-04-25T14:04:31.221187+0000","last_clean":"2024-04-25T14:04:31.221187+0000","last_became_active":"2024-04-25T14:04:30.215432+0000","last_became_peered":"2024-04-25T14:04:30.215432+0000","last_unstale":"2024-04-25T14:04:31.221187+0000","last_undegraded":"2024-04-25T14:04:31.221187+0000","last_fullsized":"2024-04-25T14:04:31.221187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215995+0000","last_change":"2024-04-25T14:04:30.211240+0000","last_active":"2024-04-25T14:04:30.215995+0000","last_peered":"2024-04-25T14:04:30.215995+0000","last_clean":"2024-04-25T14:04:30.215995+0000","last_became_active":"2024-04-25T14:04:30.210897+0000","last_became_peered":"2024-04-25T14:04:30.210897+0000","last_unstale":"2024-04-25T14:04:30.215995+0000","last_undegraded":"2024-04-25T14:04:30.215995+0000","last_fullsized":"2024-04-25T14:04:30.215995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:18:45.625324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231104+0000","last_change":"2024-04-25T14:04:30.226241+0000","last_active":"2024-04-25T14:04:30.231104+0000","last_peered":"2024-04-25T14:04:30.231104+0000","last_clean":"2024-04-25T14:04:30.231104+0000","last_became_active":"2024-04-25T14:04:30.225929+0000","last_became_peered":"2024-04-25T14:04:30.225929+0000","last_unstale":"2024-04-25T14:04:30.231104+0000","last_undegraded":"2024-04-25T14:04:30.231104+0000","last_fullsized":"2024-04-25T14:04:30.231104+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229563+0000","last_change":"2024-04-25T14:04:30.223557+0000","last_active":"2024-04-25T14:04:30.229563+0000","last_peered":"2024-04-25T14:04:30.229563+0000","last_clean":"2024-04-25T14:04:30.229563+0000","last_became_active":"2024-04-25T14:04:30.221176+0000","last_became_peered":"2024-04-25T14:04:30.221176+0000","last_unstale":"2024-04-25T14:04:30.229563+0000","last_undegraded":"2024-04-25T14:04:30.229563+0000","last_fullsized":"2024-04-25T14:04:30.229563+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:35.801013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229299+0000","last_change":"2024-04-25T14:04:30.221045+0000","last_active":"2024-04-25T14:04:30.229299+0000","last_peered":"2024-04-25T14:04:30.229299+0000","last_clean":"2024-04-25T14:04:30.229299+0000","last_became_active":"2024-04-25T14:04:30.220661+0000","last_became_peered":"2024-04-25T14:04:30.220661+0000","last_unstale":"2024-04-25T14:04:30.229299+0000","last_undegraded":"2024-04-25T14:04:30.229299+0000","last_fullsized":"2024-04-25T14:04:30.229299+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.205161+0000","last_change":"2024-04-25T14:04:30.203891+0000","last_active":"2024-04-25T14:04:30.205161+0000","last_peered":"2024-04-25T14:04:30.205161+0000","last_clean":"2024-04-25T14:04:30.205161+0000","last_became_active":"2024-04-25T14:04:30.203430+0000","last_became_peered":"2024-04-25T14:04:30.203430+0000","last_unstale":"2024-04-25T14:04:30.205161+0000","last_undegraded":"2024-04-25T14:04:30.205161+0000","last_fullsized":"2024-04-25T14:04:30.205161+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:48.736540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215641+0000","last_change":"2024-04-25T14:04:30.211972+0000","last_active":"2024-04-25T14:04:30.215641+0000","last_peered":"2024-04-25T14:04:30.215641+0000","last_clean":"2024-04-25T14:04:30.215641+0000","last_became_active":"2024-04-25T14:04:30.211695+0000","last_became_peered":"2024-04-25T14:04:30.211695+0000","last_unstale":"2024-04-25T14:04:30.215641+0000","last_undegraded":"2024-04-25T14:04:30.215641+0000","last_fullsized":"2024-04-25T14:04:30.215641+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:33.449981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231239+0000","last_change":"2024-04-25T14:04:30.227914+0000","last_active":"2024-04-25T14:04:30.231239+0000","last_peered":"2024-04-25T14:04:30.231239+0000","last_clean":"2024-04-25T14:04:30.231239+0000","last_became_active":"2024-04-25T14:04:30.227403+0000","last_became_peered":"2024-04-25T14:04:30.227403+0000","last_unstale":"2024-04-25T14:04:30.231239+0000","last_undegraded":"2024-04-25T14:04:30.231239+0000","last_fullsized":"2024-04-25T14:04:30.231239+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:04.752951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.205408+0000","last_change":"2024-04-25T14:04:30.204277+0000","last_active":"2024-04-25T14:04:30.205408+0000","last_peered":"2024-04-25T14:04:30.205408+0000","last_clean":"2024-04-25T14:04:30.205408+0000","last_became_active":"2024-04-25T14:04:30.203881+0000","last_became_peered":"2024-04-25T14:04:30.203881+0000","last_unstale":"2024-04-25T14:04:30.205408+0000","last_undegraded":"2024-04-25T14:04:30.205408+0000","last_fullsized":"2024-04-25T14:04:30.205408+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:39.227710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226642+0000","last_change":"2024-04-25T14:04:30.224171+0000","last_active":"2024-04-25T14:04:30.226642+0000","last_peered":"2024-04-25T14:04:30.226642+0000","last_clean":"2024-04-25T14:04:30.226642+0000","last_became_active":"2024-04-25T14:04:30.223883+0000","last_became_peered":"2024-04-25T14:04:30.223883+0000","last_unstale":"2024-04-25T14:04:30.226642+0000","last_undegraded":"2024-04-25T14:04:30.226642+0000","last_fullsized":"2024-04-25T14:04:30.226642+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221836+0000","last_change":"2024-04-25T14:04:30.220916+0000","last_active":"2024-04-25T14:04:31.221836+0000","last_peered":"2024-04-25T14:04:31.221836+0000","last_clean":"2024-04-25T14:04:31.221836+0000","last_became_active":"2024-04-25T14:04:30.220620+0000","last_became_peered":"2024-04-25T14:04:30.220620+0000","last_unstale":"2024-04-25T14:04:31.221836+0000","last_undegraded":"2024-04-25T14:04:31.221836+0000","last_fullsized":"2024-04-25T14:04:31.221836+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.214587+0000","last_change":"2024-04-25T14:04:30.205816+0000","last_active":"2024-04-25T14:04:30.214587+0000","last_peered":"2024-04-25T14:04:30.214587+0000","last_clean":"2024-04-25T14:04:30.214587+0000","last_became_active":"2024-04-25T14:04:30.205453+0000","last_became_peered":"2024-04-25T14:04:30.205453+0000","last_unstale":"2024-04-25T14:04:30.214587+0000","last_undegraded":"2024-04-25T14:04:30.214587+0000","last_fullsized":"2024-04-25T14:04:30.214587+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:02.742205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224044+0000","last_change":"2024-04-25T14:04:30.218132+0000","last_active":"2024-04-25T14:04:30.224044+0000","last_peered":"2024-04-25T14:04:30.224044+0000","last_clean":"2024-04-25T14:04:30.224044+0000","last_became_active":"2024-04-25T14:04:30.217293+0000","last_became_peered":"2024-04-25T14:04:30.217293+0000","last_unstale":"2024-04-25T14:04:30.224044+0000","last_undegraded":"2024-04-25T14:04:30.224044+0000","last_fullsized":"2024-04-25T14:04:30.224044+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:40.569444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222395+0000","last_change":"2024-04-25T14:04:30.218172+0000","last_active":"2024-04-25T14:04:31.222395+0000","last_peered":"2024-04-25T14:04:31.222395+0000","last_clean":"2024-04-25T14:04:31.222395+0000","last_became_active":"2024-04-25T14:04:30.217788+0000","last_became_peered":"2024-04-25T14:04:30.217788+0000","last_unstale":"2024-04-25T14:04:31.222395+0000","last_undegraded":"2024-04-25T14:04:31.222395+0000","last_fullsized":"2024-04-25T14:04:31.222395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.227288+0000","last_change":"2024-04-25T14:04:30.223862+0000","last_active":"2024-04-25T14:04:30.227288+0000","last_peered":"2024-04-25T14:04:30.227288+0000","last_clean":"2024-04-25T14:04:30.227288+0000","last_became_active":"2024-04-25T14:04:30.223569+0000","last_became_peered":"2024-04-25T14:04:30.223569+0000","last_unstale":"2024-04-25T14:04:30.227288+0000","last_undegraded":"2024-04-25T14:04:30.227288+0000","last_fullsized":"2024-04-25T14:04:30.227288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219951+0000","last_change":"2024-04-25T14:04:30.220878+0000","last_active":"2024-04-25T14:04:31.219951+0000","last_peered":"2024-04-25T14:04:31.219951+0000","last_clean":"2024-04-25T14:04:31.219951+0000","last_became_active":"2024-04-25T14:04:30.220575+0000","last_became_peered":"2024-04-25T14:04:30.220575+0000","last_unstale":"2024-04-25T14:04:31.219951+0000","last_undegraded":"2024-04-25T14:04:31.219951+0000","last_fullsized":"2024-04-25T14:04:31.219951+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220438+0000","last_change":"2024-04-25T14:04:30.224581+0000","last_active":"2024-04-25T14:04:31.220438+0000","last_peered":"2024-04-25T14:04:31.220438+0000","last_clean":"2024-04-25T14:04:31.220438+0000","last_became_active":"2024-04-25T14:04:30.224277+0000","last_became_peered":"2024-04-25T14:04:30.224277+0000","last_unstale":"2024-04-25T14:04:31.220438+0000","last_undegraded":"2024-04-25T14:04:31.220438+0000","last_fullsized":"2024-04-25T14:04:31.220438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:10.900532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231225+0000","last_change":"2024-04-25T14:04:30.228708+0000","last_active":"2024-04-25T14:04:30.231225+0000","last_peered":"2024-04-25T14:04:30.231225+0000","last_clean":"2024-04-25T14:04:30.231225+0000","last_became_active":"2024-04-25T14:04:30.228416+0000","last_became_peered":"2024-04-25T14:04:30.228416+0000","last_unstale":"2024-04-25T14:04:30.231225+0000","last_undegraded":"2024-04-25T14:04:30.231225+0000","last_fullsized":"2024-04-25T14:04:30.231225+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:42.490744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222467+0000","last_change":"2024-04-25T14:04:30.215506+0000","last_active":"2024-04-25T14:04:30.222467+0000","last_peered":"2024-04-25T14:04:30.222467+0000","last_clean":"2024-04-25T14:04:30.222467+0000","last_became_active":"2024-04-25T14:04:30.215222+0000","last_became_peered":"2024-04-25T14:04:30.215222+0000","last_unstale":"2024-04-25T14:04:30.222467+0000","last_undegraded":"2024-04-25T14:04:30.222467+0000","last_fullsized":"2024-04-25T14:04:30.222467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:02.964930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225126+0000","last_change":"2024-04-25T14:04:30.218465+0000","last_active":"2024-04-25T14:04:30.225126+0000","last_peered":"2024-04-25T14:04:30.225126+0000","last_clean":"2024-04-25T14:04:30.225126+0000","last_became_active":"2024-04-25T14:04:30.218179+0000","last_became_peered":"2024-04-25T14:04:30.218179+0000","last_unstale":"2024-04-25T14:04:30.225126+0000","last_undegraded":"2024-04-25T14:04:30.225126+0000","last_fullsized":"2024-04-25T14:04:30.225126+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.794183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225327+0000","last_change":"2024-04-25T14:04:30.222735+0000","last_active":"2024-04-25T14:04:30.225327+0000","last_peered":"2024-04-25T14:04:30.225327+0000","last_clean":"2024-04-25T14:04:30.225327+0000","last_became_active":"2024-04-25T14:04:30.221730+0000","last_became_peered":"2024-04-25T14:04:30.221730+0000","last_unstale":"2024-04-25T14:04:30.225327+0000","last_undegraded":"2024-04-25T14:04:30.225327+0000","last_fullsized":"2024-04-25T14:04:30.225327+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:13.981930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216593+0000","last_change":"2024-04-25T14:04:30.210180+0000","last_active":"2024-04-25T14:04:30.216593+0000","last_peered":"2024-04-25T14:04:30.216593+0000","last_clean":"2024-04-25T14:04:30.216593+0000","last_became_active":"2024-04-25T14:04:30.209890+0000","last_became_peered":"2024-04-25T14:04:30.209890+0000","last_unstale":"2024-04-25T14:04:30.216593+0000","last_undegraded":"2024-04-25T14:04:30.216593+0000","last_fullsized":"2024-04-25T14:04:30.216593+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231177+0000","last_change":"2024-04-25T14:04:30.227883+0000","last_active":"2024-04-25T14:04:30.231177+0000","last_peered":"2024-04-25T14:04:30.231177+0000","last_clean":"2024-04-25T14:04:30.231177+0000","last_became_active":"2024-04-25T14:04:30.227598+0000","last_became_peered":"2024-04-25T14:04:30.227598+0000","last_unstale":"2024-04-25T14:04:30.231177+0000","last_undegraded":"2024-04-25T14:04:30.231177+0000","last_fullsized":"2024-04-25T14:04:30.231177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:10.900532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220172+0000","last_change":"2024-04-25T14:04:30.218772+0000","last_active":"2024-04-25T14:04:31.220172+0000","last_peered":"2024-04-25T14:04:31.220172+0000","last_clean":"2024-04-25T14:04:31.220172+0000","last_became_active":"2024-04-25T14:04:30.218475+0000","last_became_peered":"2024-04-25T14:04:30.218475+0000","last_unstale":"2024-04-25T14:04:31.220172+0000","last_undegraded":"2024-04-25T14:04:31.220172+0000","last_fullsized":"2024-04-25T14:04:31.220172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:45.046031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231835+0000","last_change":"2024-04-25T14:04:30.230664+0000","last_active":"2024-04-25T14:04:30.231835+0000","last_peered":"2024-04-25T14:04:30.231835+0000","last_clean":"2024-04-25T14:04:30.231835+0000","last_became_active":"2024-04-25T14:04:30.229801+0000","last_became_peered":"2024-04-25T14:04:30.229801+0000","last_unstale":"2024-04-25T14:04:30.231835+0000","last_undegraded":"2024-04-25T14:04:30.231835+0000","last_fullsized":"2024-04-25T14:04:30.231835+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:35.801013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222395+0000","last_change":"2024-04-25T14:04:30.214418+0000","last_active":"2024-04-25T14:04:31.222395+0000","last_peered":"2024-04-25T14:04:31.222395+0000","last_clean":"2024-04-25T14:04:31.222395+0000","last_became_active":"2024-04-25T14:04:30.214126+0000","last_became_peered":"2024-04-25T14:04:30.214126+0000","last_unstale":"2024-04-25T14:04:31.222395+0000","last_undegraded":"2024-04-25T14:04:31.222395+0000","last_fullsized":"2024-04-25T14:04:31.222395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226671+0000","last_change":"2024-04-25T14:04:30.223706+0000","last_active":"2024-04-25T14:04:30.226671+0000","last_peered":"2024-04-25T14:04:30.226671+0000","last_clean":"2024-04-25T14:04:30.226671+0000","last_became_active":"2024-04-25T14:04:30.223403+0000","last_became_peered":"2024-04-25T14:04:30.223403+0000","last_unstale":"2024-04-25T14:04:30.226671+0000","last_undegraded":"2024-04-25T14:04:30.226671+0000","last_fullsized":"2024-04-25T14:04:30.226671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226144+0000","last_change":"2024-04-25T14:04:30.222373+0000","last_active":"2024-04-25T14:04:30.226144+0000","last_peered":"2024-04-25T14:04:30.226144+0000","last_clean":"2024-04-25T14:04:30.226144+0000","last_became_active":"2024-04-25T14:04:30.221976+0000","last_became_peered":"2024-04-25T14:04:30.221976+0000","last_unstale":"2024-04-25T14:04:30.226144+0000","last_undegraded":"2024-04-25T14:04:30.226144+0000","last_fullsized":"2024-04-25T14:04:30.226144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:13.981930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219925+0000","last_change":"2024-04-25T14:04:30.218727+0000","last_active":"2024-04-25T14:04:31.219925+0000","last_peered":"2024-04-25T14:04:31.219925+0000","last_clean":"2024-04-25T14:04:31.219925+0000","last_became_active":"2024-04-25T14:04:30.218414+0000","last_became_peered":"2024-04-25T14:04:30.218414+0000","last_unstale":"2024-04-25T14:04:31.219925+0000","last_undegraded":"2024-04-25T14:04:31.219925+0000","last_fullsized":"2024-04-25T14:04:31.219925+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220565+0000","last_change":"2024-04-25T14:04:29.212267+0000","last_active":"2024-04-25T14:04:31.220565+0000","last_peered":"2024-04-25T14:04:31.220565+0000","last_clean":"2024-04-25T14:04:31.220565+0000","last_became_active":"2024-04-25T14:04:29.211937+0000","last_became_peered":"2024-04-25T14:04:29.211937+0000","last_unstale":"2024-04-25T14:04:31.220565+0000","last_undegraded":"2024-04-25T14:04:31.220565+0000","last_fullsized":"2024-04-25T14:04:31.220565+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:23.626937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231252+0000","last_change":"2024-04-25T14:04:30.223856+0000","last_active":"2024-04-25T14:04:30.231252+0000","last_peered":"2024-04-25T14:04:30.231252+0000","last_clean":"2024-04-25T14:04:30.231252+0000","last_became_active":"2024-04-25T14:04:30.223557+0000","last_became_peered":"2024-04-25T14:04:30.223557+0000","last_unstale":"2024-04-25T14:04:30.231252+0000","last_undegraded":"2024-04-25T14:04:30.231252+0000","last_fullsized":"2024-04-25T14:04:30.231252+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229955+0000","last_change":"2024-04-25T14:04:29.217418+0000","last_active":"2024-04-25T14:04:30.229955+0000","last_peered":"2024-04-25T14:04:30.229955+0000","last_clean":"2024-04-25T14:04:30.229955+0000","last_became_active":"2024-04-25T14:04:29.216344+0000","last_became_peered":"2024-04-25T14:04:29.216344+0000","last_unstale":"2024-04-25T14:04:30.229955+0000","last_undegraded":"2024-04-25T14:04:30.229955+0000","last_fullsized":"2024-04-25T14:04:30.229955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:42:20.310590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221300+0000","last_change":"2024-04-25T14:04:30.223981+0000","last_active":"2024-04-25T14:04:31.221300+0000","last_peered":"2024-04-25T14:04:31.221300+0000","last_clean":"2024-04-25T14:04:31.221300+0000","last_became_active":"2024-04-25T14:04:30.223463+0000","last_became_peered":"2024-04-25T14:04:30.223463+0000","last_unstale":"2024-04-25T14:04:31.221300+0000","last_undegraded":"2024-04-25T14:04:31.221300+0000","last_fullsized":"2024-04-25T14:04:31.221300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:16.408377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224876+0000","last_change":"2024-04-25T14:04:29.211979+0000","last_active":"2024-04-25T14:04:30.224876+0000","last_peered":"2024-04-25T14:04:30.224876+0000","last_clean":"2024-04-25T14:04:30.224876+0000","last_became_active":"2024-04-25T14:04:29.211471+0000","last_became_peered":"2024-04-25T14:04:29.211471+0000","last_unstale":"2024-04-25T14:04:30.224876+0000","last_undegraded":"2024-04-25T14:04:30.224876+0000","last_fullsized":"2024-04-25T14:04:30.224876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:01.076515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224979+0000","last_change":"2024-04-25T14:04:30.209984+0000","last_active":"2024-04-25T14:04:30.224979+0000","last_peered":"2024-04-25T14:04:30.224979+0000","last_clean":"2024-04-25T14:04:30.224979+0000","last_became_active":"2024-04-25T14:04:30.209645+0000","last_became_peered":"2024-04-25T14:04:30.209645+0000","last_unstale":"2024-04-25T14:04:30.224979+0000","last_undegraded":"2024-04-25T14:04:30.224979+0000","last_fullsized":"2024-04-25T14:04:30.224979+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231441+0000","last_change":"2024-04-25T14:04:29.213802+0000","last_active":"2024-04-25T14:04:30.231441+0000","last_peered":"2024-04-25T14:04:30.231441+0000","last_clean":"2024-04-25T14:04:30.231441+0000","last_became_active":"2024-04-25T14:04:29.213479+0000","last_became_peered":"2024-04-25T14:04:29.213479+0000","last_unstale":"2024-04-25T14:04:30.231441+0000","last_undegraded":"2024-04-25T14:04:30.231441+0000","last_fullsized":"2024-04-25T14:04:30.231441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:39.567458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221328+0000","last_change":"2024-04-25T14:04:30.217870+0000","last_active":"2024-04-25T14:04:31.221328+0000","last_peered":"2024-04-25T14:04:31.221328+0000","last_clean":"2024-04-25T14:04:31.221328+0000","last_became_active":"2024-04-25T14:04:30.217577+0000","last_became_peered":"2024-04-25T14:04:30.217577+0000","last_unstale":"2024-04-25T14:04:31.221328+0000","last_undegraded":"2024-04-25T14:04:31.221328+0000","last_fullsized":"2024-04-25T14:04:31.221328+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225103+0000","last_change":"2024-04-25T14:04:29.221871+0000","last_active":"2024-04-25T14:04:30.225103+0000","last_peered":"2024-04-25T14:04:30.225103+0000","last_clean":"2024-04-25T14:04:30.225103+0000","last_became_active":"2024-04-25T14:04:29.221524+0000","last_became_peered":"2024-04-25T14:04:29.221524+0000","last_unstale":"2024-04-25T14:04:30.225103+0000","last_undegraded":"2024-04-25T14:04:30.225103+0000","last_fullsized":"2024-04-25T14:04:30.225103+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:15.406391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220373+0000","last_change":"2024-04-25T14:04:30.218754+0000","last_active":"2024-04-25T14:04:31.220373+0000","last_peered":"2024-04-25T14:04:31.220373+0000","last_clean":"2024-04-25T14:04:31.220373+0000","last_became_active":"2024-04-25T14:04:30.218448+0000","last_became_peered":"2024-04-25T14:04:30.218448+0000","last_unstale":"2024-04-25T14:04:31.220373+0000","last_undegraded":"2024-04-25T14:04:31.220373+0000","last_fullsized":"2024-04-25T14:04:31.220373+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221434+0000","last_change":"2024-04-25T14:04:29.212086+0000","last_active":"2024-04-25T14:04:31.221434+0000","last_peered":"2024-04-25T14:04:31.221434+0000","last_clean":"2024-04-25T14:04:31.221434+0000","last_became_active":"2024-04-25T14:04:29.211254+0000","last_became_peered":"2024-04-25T14:04:29.211254+0000","last_unstale":"2024-04-25T14:04:31.221434+0000","last_undegraded":"2024-04-25T14:04:31.221434+0000","last_fullsized":"2024-04-25T14:04:31.221434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216620+0000","last_change":"2024-04-25T14:04:30.210445+0000","last_active":"2024-04-25T14:04:30.216620+0000","last_peered":"2024-04-25T14:04:30.216620+0000","last_clean":"2024-04-25T14:04:30.216620+0000","last_became_active":"2024-04-25T14:04:30.209874+0000","last_became_peered":"2024-04-25T14:04:30.209874+0000","last_unstale":"2024-04-25T14:04:30.216620+0000","last_undegraded":"2024-04-25T14:04:30.216620+0000","last_fullsized":"2024-04-25T14:04:30.216620+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220685+0000","last_change":"2024-04-25T14:04:29.208335+0000","last_active":"2024-04-25T14:04:31.220685+0000","last_peered":"2024-04-25T14:04:31.220685+0000","last_clean":"2024-04-25T14:04:31.220685+0000","last_became_active":"2024-04-25T14:04:29.207928+0000","last_became_peered":"2024-04-25T14:04:29.207928+0000","last_unstale":"2024-04-25T14:04:31.220685+0000","last_undegraded":"2024-04-25T14:04:31.220685+0000","last_fullsized":"2024-04-25T14:04:31.220685+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:49.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220355+0000","last_change":"2024-04-25T14:04:30.222225+0000","last_active":"2024-04-25T14:04:31.220355+0000","last_peered":"2024-04-25T14:04:31.220355+0000","last_clean":"2024-04-25T14:04:31.220355+0000","last_became_active":"2024-04-25T14:04:30.221901+0000","last_became_peered":"2024-04-25T14:04:30.221901+0000","last_unstale":"2024-04-25T14:04:31.220355+0000","last_undegraded":"2024-04-25T14:04:31.220355+0000","last_fullsized":"2024-04-25T14:04:31.220355+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.208221+0000","last_change":"2024-04-25T14:04:29.217227+0000","last_active":"2024-04-25T14:04:30.208221+0000","last_peered":"2024-04-25T14:04:30.208221+0000","last_clean":"2024-04-25T14:04:30.208221+0000","last_became_active":"2024-04-25T14:04:29.216904+0000","last_became_peered":"2024-04-25T14:04:29.216904+0000","last_unstale":"2024-04-25T14:04:30.208221+0000","last_undegraded":"2024-04-25T14:04:30.208221+0000","last_fullsized":"2024-04-25T14:04:30.208221+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:44.044045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220879+0000","last_change":"2024-04-25T14:04:30.219118+0000","last_active":"2024-04-25T14:04:31.220879+0000","last_peered":"2024-04-25T14:04:31.220879+0000","last_clean":"2024-04-25T14:04:31.220879+0000","last_became_active":"2024-04-25T14:04:30.214059+0000","last_became_peered":"2024-04-25T14:04:30.214059+0000","last_unstale":"2024-04-25T14:04:31.220879+0000","last_undegraded":"2024-04-25T14:04:31.220879+0000","last_fullsized":"2024-04-25T14:04:31.220879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:25.133215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220277+0000","last_change":"2024-04-25T14:04:29.209016+0000","last_active":"2024-04-25T14:04:31.220277+0000","last_peered":"2024-04-25T14:04:31.220277+0000","last_clean":"2024-04-25T14:04:31.220277+0000","last_became_active":"2024-04-25T14:04:29.208687+0000","last_became_peered":"2024-04-25T14:04:29.208687+0000","last_unstale":"2024-04-25T14:04:31.220277+0000","last_undegraded":"2024-04-25T14:04:31.220277+0000","last_fullsized":"2024-04-25T14:04:31.220277+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:00.939721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.218186+0000","last_change":"2024-04-25T14:04:30.216937+0000","last_active":"2024-04-25T14:04:30.218186+0000","last_peered":"2024-04-25T14:04:30.218186+0000","last_clean":"2024-04-25T14:04:30.218186+0000","last_became_active":"2024-04-25T14:04:30.216609+0000","last_became_peered":"2024-04-25T14:04:30.216609+0000","last_unstale":"2024-04-25T14:04:30.218186+0000","last_undegraded":"2024-04-25T14:04:30.218186+0000","last_fullsized":"2024-04-25T14:04:30.218186+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:17.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221343+0000","last_change":"2024-04-25T14:04:29.221582+0000","last_active":"2024-04-25T14:04:31.221343+0000","last_peered":"2024-04-25T14:04:31.221343+0000","last_clean":"2024-04-25T14:04:31.221343+0000","last_became_active":"2024-04-25T14:04:29.221337+0000","last_became_peered":"2024-04-25T14:04:29.221337+0000","last_unstale":"2024-04-25T14:04:31.221343+0000","last_undegraded":"2024-04-25T14:04:31.221343+0000","last_fullsized":"2024-04-25T14:04:31.221343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:51.800914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231767+0000","last_change":"2024-04-25T14:04:30.229612+0000","last_active":"2024-04-25T14:04:30.231767+0000","last_peered":"2024-04-25T14:04:30.231767+0000","last_clean":"2024-04-25T14:04:30.231767+0000","last_became_active":"2024-04-25T14:04:30.229322+0000","last_became_peered":"2024-04-25T14:04:30.229322+0000","last_unstale":"2024-04-25T14:04:30.231767+0000","last_undegraded":"2024-04-25T14:04:30.231767+0000","last_fullsized":"2024-04-25T14:04:30.231767+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220592+0000","last_change":"2024-04-25T14:04:29.211807+0000","last_active":"2024-04-25T14:04:31.220592+0000","last_peered":"2024-04-25T14:04:31.220592+0000","last_clean":"2024-04-25T14:04:31.220592+0000","last_became_active":"2024-04-25T14:04:29.211462+0000","last_became_peered":"2024-04-25T14:04:29.211462+0000","last_unstale":"2024-04-25T14:04:31.220592+0000","last_undegraded":"2024-04-25T14:04:31.220592+0000","last_fullsized":"2024-04-25T14:04:31.220592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:09.898546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221449+0000","last_change":"2024-04-25T14:04:30.207374+0000","last_active":"2024-04-25T14:04:31.221449+0000","last_peered":"2024-04-25T14:04:31.221449+0000","last_clean":"2024-04-25T14:04:31.221449+0000","last_became_active":"2024-04-25T14:04:30.207025+0000","last_became_peered":"2024-04-25T14:04:30.207025+0000","last_unstale":"2024-04-25T14:04:31.221449+0000","last_undegraded":"2024-04-25T14:04:31.221449+0000","last_fullsized":"2024-04-25T14:04:31.221449+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"55'6","reported_seq":18,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.234034+0000","last_change":"2024-04-25T14:04:29.219922+0000","last_active":"2024-04-25T14:04:30.234034+0000","last_peered":"2024-04-25T14:04:30.234034+0000","last_clean":"2024-04-25T14:04:30.234034+0000","last_became_active":"2024-04-25T14:04:29.219622+0000","last_became_peered":"2024-04-25T14:04:29.219622+0000","last_unstale":"2024-04-25T14:04:30.234034+0000","last_undegraded":"2024-04-25T14:04:30.234034+0000","last_fullsized":"2024-04-25T14:04:30.234034+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:32.187632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.220041+0000","last_change":"2024-04-25T14:04:30.217376+0000","last_active":"2024-04-25T14:04:30.220041+0000","last_peered":"2024-04-25T14:04:30.220041+0000","last_clean":"2024-04-25T14:04:30.220041+0000","last_became_active":"2024-04-25T14:04:30.217039+0000","last_became_peered":"2024-04-25T14:04:30.217039+0000","last_unstale":"2024-04-25T14:04:30.220041+0000","last_undegraded":"2024-04-25T14:04:30.220041+0000","last_fullsized":"2024-04-25T14:04:30.220041+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:33.189618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225214+0000","last_change":"2024-04-25T14:04:29.214433+0000","last_active":"2024-04-25T14:04:30.225214+0000","last_peered":"2024-04-25T14:04:30.225214+0000","last_clean":"2024-04-25T14:04:30.225214+0000","last_became_active":"2024-04-25T14:04:29.214086+0000","last_became_peered":"2024-04-25T14:04:29.214086+0000","last_unstale":"2024-04-25T14:04:30.225214+0000","last_undegraded":"2024-04-25T14:04:30.225214+0000","last_fullsized":"2024-04-25T14:04:30.225214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:00.939721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224074+0000","last_change":"2024-04-25T14:04:30.218458+0000","last_active":"2024-04-25T14:04:30.224074+0000","last_peered":"2024-04-25T14:04:30.224074+0000","last_clean":"2024-04-25T14:04:30.224074+0000","last_became_active":"2024-04-25T14:04:30.218145+0000","last_became_peered":"2024-04-25T14:04:30.218145+0000","last_unstale":"2024-04-25T14:04:30.224074+0000","last_undegraded":"2024-04-25T14:04:30.224074+0000","last_fullsized":"2024-04-25T14:04:30.224074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.917949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221060+0000","last_change":"2024-04-25T14:04:29.222899+0000","last_active":"2024-04-25T14:04:31.221060+0000","last_peered":"2024-04-25T14:04:31.221060+0000","last_clean":"2024-04-25T14:04:31.221060+0000","last_became_active":"2024-04-25T14:04:29.222626+0000","last_became_peered":"2024-04-25T14:04:29.222626+0000","last_unstale":"2024-04-25T14:04:31.221060+0000","last_undegraded":"2024-04-25T14:04:31.221060+0000","last_fullsized":"2024-04-25T14:04:31.221060+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:11.326588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229744+0000","last_change":"2024-04-25T14:04:30.219657+0000","last_active":"2024-04-25T14:04:30.229744+0000","last_peered":"2024-04-25T14:04:30.229744+0000","last_clean":"2024-04-25T14:04:30.229744+0000","last_became_active":"2024-04-25T14:04:30.219241+0000","last_became_peered":"2024-04-25T14:04:30.219241+0000","last_unstale":"2024-04-25T14:04:30.229744+0000","last_undegraded":"2024-04-25T14:04:30.229744+0000","last_fullsized":"2024-04-25T14:04:30.229744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221934+0000","last_change":"2024-04-25T14:04:29.208196+0000","last_active":"2024-04-25T14:04:31.221934+0000","last_peered":"2024-04-25T14:04:31.221934+0000","last_clean":"2024-04-25T14:04:31.221934+0000","last_became_active":"2024-04-25T14:04:29.207877+0000","last_became_peered":"2024-04-25T14:04:29.207877+0000","last_unstale":"2024-04-25T14:04:31.221934+0000","last_undegraded":"2024-04-25T14:04:31.221934+0000","last_fullsized":"2024-04-25T14:04:31.221934+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:44.044045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225296+0000","last_change":"2024-04-25T14:04:30.218608+0000","last_active":"2024-04-25T14:04:30.225296+0000","last_peered":"2024-04-25T14:04:30.225296+0000","last_clean":"2024-04-25T14:04:30.225296+0000","last_became_active":"2024-04-25T14:04:30.217690+0000","last_became_peered":"2024-04-25T14:04:30.217690+0000","last_unstale":"2024-04-25T14:04:30.225296+0000","last_undegraded":"2024-04-25T14:04:30.225296+0000","last_fullsized":"2024-04-25T14:04:30.225296+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:17.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"55'8","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.232106+0000","last_change":"2024-04-25T14:04:29.219053+0000","last_active":"2024-04-25T14:04:30.232106+0000","last_peered":"2024-04-25T14:04:30.232106+0000","last_clean":"2024-04-25T14:04:30.232106+0000","last_became_active":"2024-04-25T14:04:29.217576+0000","last_became_peered":"2024-04-25T14:04:29.217576+0000","last_unstale":"2024-04-25T14:04:30.232106+0000","last_undegraded":"2024-04-25T14:04:30.232106+0000","last_fullsized":"2024-04-25T14:04:30.232106+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.792198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229369+0000","last_change":"2024-04-25T14:04:30.218916+0000","last_active":"2024-04-25T14:04:30.229369+0000","last_peered":"2024-04-25T14:04:30.229369+0000","last_clean":"2024-04-25T14:04:30.229369+0000","last_became_active":"2024-04-25T14:04:30.218466+0000","last_became_peered":"2024-04-25T14:04:30.218466+0000","last_unstale":"2024-04-25T14:04:30.229369+0000","last_undegraded":"2024-04-25T14:04:30.229369+0000","last_fullsized":"2024-04-25T14:04:30.229369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.230228+0000","last_change":"2024-04-25T14:04:29.220594+0000","last_active":"2024-04-25T14:04:30.230228+0000","last_peered":"2024-04-25T14:04:30.230228+0000","last_clean":"2024-04-25T14:04:30.230228+0000","last_became_active":"2024-04-25T14:04:29.220302+0000","last_became_peered":"2024-04-25T14:04:29.220302+0000","last_unstale":"2024-04-25T14:04:30.230228+0000","last_undegraded":"2024-04-25T14:04:30.230228+0000","last_fullsized":"2024-04-25T14:04:30.230228+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:09.898546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231186+0000","last_change":"2024-04-25T14:04:30.227183+0000","last_active":"2024-04-25T14:04:30.231186+0000","last_peered":"2024-04-25T14:04:30.231186+0000","last_clean":"2024-04-25T14:04:30.231186+0000","last_became_active":"2024-04-25T14:04:30.226888+0000","last_became_peered":"2024-04-25T14:04:30.226888+0000","last_unstale":"2024-04-25T14:04:30.231186+0000","last_undegraded":"2024-04-25T14:04:30.231186+0000","last_fullsized":"2024-04-25T14:04:30.231186+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:45.046031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216930+0000","last_change":"2024-04-25T14:04:29.206123+0000","last_active":"2024-04-25T14:04:30.216930+0000","last_peered":"2024-04-25T14:04:30.216930+0000","last_clean":"2024-04-25T14:04:30.216930+0000","last_became_active":"2024-04-25T14:04:29.205802+0000","last_became_peered":"2024-04-25T14:04:29.205802+0000","last_unstale":"2024-04-25T14:04:30.216930+0000","last_undegraded":"2024-04-25T14:04:30.216930+0000","last_fullsized":"2024-04-25T14:04:30.216930+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:25.192007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220951+0000","last_change":"2024-04-25T14:04:30.226936+0000","last_active":"2024-04-25T14:04:31.220951+0000","last_peered":"2024-04-25T14:04:31.220951+0000","last_clean":"2024-04-25T14:04:31.220951+0000","last_became_active":"2024-04-25T14:04:30.226644+0000","last_became_peered":"2024-04-25T14:04:30.226644+0000","last_unstale":"2024-04-25T14:04:31.220951+0000","last_undegraded":"2024-04-25T14:04:31.220951+0000","last_fullsized":"2024-04-25T14:04:31.220951+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:18:45.625324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220791+0000","last_change":"2024-04-25T14:04:29.213637+0000","last_active":"2024-04-25T14:04:31.220791+0000","last_peered":"2024-04-25T14:04:31.220791+0000","last_clean":"2024-04-25T14:04:31.220791+0000","last_became_active":"2024-04-25T14:04:29.213288+0000","last_became_peered":"2024-04-25T14:04:29.213288+0000","last_unstale":"2024-04-25T14:04:31.220791+0000","last_undegraded":"2024-04-25T14:04:31.220791+0000","last_fullsized":"2024-04-25T14:04:31.220791+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:45:09.681772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222600+0000","last_change":"2024-04-25T14:04:30.220608+0000","last_active":"2024-04-25T14:04:30.222600+0000","last_peered":"2024-04-25T14:04:30.222600+0000","last_clean":"2024-04-25T14:04:30.222600+0000","last_became_active":"2024-04-25T14:04:30.220299+0000","last_became_peered":"2024-04-25T14:04:30.220299+0000","last_unstale":"2024-04-25T14:04:30.222600+0000","last_undegraded":"2024-04-25T14:04:30.222600+0000","last_fullsized":"2024-04-25T14:04:30.222600+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221152+0000","last_change":"2024-04-25T14:04:29.215173+0000","last_active":"2024-04-25T14:04:31.221152+0000","last_peered":"2024-04-25T14:04:31.221152+0000","last_clean":"2024-04-25T14:04:31.221152+0000","last_became_active":"2024-04-25T14:04:29.214864+0000","last_became_peered":"2024-04-25T14:04:29.214864+0000","last_unstale":"2024-04-25T14:04:31.221152+0000","last_undegraded":"2024-04-25T14:04:31.221152+0000","last_fullsized":"2024-04-25T14:04:31.221152+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231257+0000","last_change":"2024-04-25T14:04:30.223667+0000","last_active":"2024-04-25T14:04:30.231257+0000","last_peered":"2024-04-25T14:04:30.231257+0000","last_clean":"2024-04-25T14:04:30.231257+0000","last_became_active":"2024-04-25T14:04:30.223339+0000","last_became_peered":"2024-04-25T14:04:30.223339+0000","last_unstale":"2024-04-25T14:04:30.231257+0000","last_undegraded":"2024-04-25T14:04:30.231257+0000","last_fullsized":"2024-04-25T14:04:30.231257+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224655+0000","last_change":"2024-04-25T14:04:29.212470+0000","last_active":"2024-04-25T14:04:30.224655+0000","last_peered":"2024-04-25T14:04:30.224655+0000","last_clean":"2024-04-25T14:04:30.224655+0000","last_became_active":"2024-04-25T14:04:29.212178+0000","last_became_peered":"2024-04-25T14:04:29.212178+0000","last_unstale":"2024-04-25T14:04:30.224655+0000","last_undegraded":"2024-04-25T14:04:30.224655+0000","last_fullsized":"2024-04-25T14:04:30.224655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220014+0000","last_change":"2024-04-25T14:04:30.223666+0000","last_active":"2024-04-25T14:04:31.220014+0000","last_peered":"2024-04-25T14:04:31.220014+0000","last_clean":"2024-04-25T14:04:31.220014+0000","last_became_active":"2024-04-25T14:04:30.223371+0000","last_became_peered":"2024-04-25T14:04:30.223371+0000","last_unstale":"2024-04-25T14:04:31.220014+0000","last_undegraded":"2024-04-25T14:04:31.220014+0000","last_fullsized":"2024-04-25T14:04:31.220014+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203144+0000","last_change":"2024-04-25T14:04:29.219331+0000","last_active":"2024-04-25T14:04:30.203144+0000","last_peered":"2024-04-25T14:04:30.203144+0000","last_clean":"2024-04-25T14:04:30.203144+0000","last_became_active":"2024-04-25T14:04:29.219039+0000","last_became_peered":"2024-04-25T14:04:29.219039+0000","last_unstale":"2024-04-25T14:04:30.203144+0000","last_undegraded":"2024-04-25T14:04:30.203144+0000","last_fullsized":"2024-04-25T14:04:30.203144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:28.674456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231828+0000","last_change":"2024-04-25T14:04:30.230603+0000","last_active":"2024-04-25T14:04:30.231828+0000","last_peered":"2024-04-25T14:04:30.231828+0000","last_clean":"2024-04-25T14:04:30.231828+0000","last_became_active":"2024-04-25T14:04:30.230111+0000","last_became_peered":"2024-04-25T14:04:30.230111+0000","last_unstale":"2024-04-25T14:04:30.231828+0000","last_undegraded":"2024-04-25T14:04:30.231828+0000","last_fullsized":"2024-04-25T14:04:30.231828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222328+0000","last_change":"2024-04-25T14:04:29.217705+0000","last_active":"2024-04-25T14:04:30.222328+0000","last_peered":"2024-04-25T14:04:30.222328+0000","last_clean":"2024-04-25T14:04:30.222328+0000","last_became_active":"2024-04-25T14:04:29.217399+0000","last_became_peered":"2024-04-25T14:04:29.217399+0000","last_unstale":"2024-04-25T14:04:30.222328+0000","last_undegraded":"2024-04-25T14:04:30.222328+0000","last_fullsized":"2024-04-25T14:04:30.222328+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:28.674456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216029+0000","last_change":"2024-04-25T14:04:30.212096+0000","last_active":"2024-04-25T14:04:30.216029+0000","last_peered":"2024-04-25T14:04:30.216029+0000","last_clean":"2024-04-25T14:04:30.216029+0000","last_became_active":"2024-04-25T14:04:30.211801+0000","last_became_peered":"2024-04-25T14:04:30.211801+0000","last_unstale":"2024-04-25T14:04:30.216029+0000","last_undegraded":"2024-04-25T14:04:30.216029+0000","last_fullsized":"2024-04-25T14:04:30.216029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203330+0000","last_change":"2024-04-25T14:04:29.209296+0000","last_active":"2024-04-25T14:04:30.203330+0000","last_peered":"2024-04-25T14:04:30.203330+0000","last_clean":"2024-04-25T14:04:30.203330+0000","last_became_active":"2024-04-25T14:04:29.208978+0000","last_became_peered":"2024-04-25T14:04:29.208978+0000","last_unstale":"2024-04-25T14:04:30.203330+0000","last_undegraded":"2024-04-25T14:04:30.203330+0000","last_fullsized":"2024-04-25T14:04:30.203330+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:34.634391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.217192+0000","last_change":"2024-04-25T14:04:30.216153+0000","last_active":"2024-04-25T14:04:30.217192+0000","last_peered":"2024-04-25T14:04:30.217192+0000","last_clean":"2024-04-25T14:04:30.217192+0000","last_became_active":"2024-04-25T14:04:30.211851+0000","last_became_peered":"2024-04-25T14:04:30.211851+0000","last_unstale":"2024-04-25T14:04:30.217192+0000","last_undegraded":"2024-04-25T14:04:30.217192+0000","last_fullsized":"2024-04-25T14:04:30.217192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.794183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220898+0000","last_change":"2024-04-25T14:04:29.223011+0000","last_active":"2024-04-25T14:04:31.220898+0000","last_peered":"2024-04-25T14:04:31.220898+0000","last_clean":"2024-04-25T14:04:31.220898+0000","last_became_active":"2024-04-25T14:04:29.222677+0000","last_became_peered":"2024-04-25T14:04:29.222677+0000","last_unstale":"2024-04-25T14:04:31.220898+0000","last_undegraded":"2024-04-25T14:04:31.220898+0000","last_fullsized":"2024-04-25T14:04:31.220898+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:51.800914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215967+0000","last_change":"2024-04-25T14:04:30.206536+0000","last_active":"2024-04-25T14:04:30.215967+0000","last_peered":"2024-04-25T14:04:30.215967+0000","last_clean":"2024-04-25T14:04:30.215967+0000","last_became_active":"2024-04-25T14:04:30.206254+0000","last_became_peered":"2024-04-25T14:04:30.206254+0000","last_unstale":"2024-04-25T14:04:30.215967+0000","last_undegraded":"2024-04-25T14:04:30.215967+0000","last_fullsized":"2024-04-25T14:04:30.215967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:25.133215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220710+0000","last_change":"2024-04-25T14:04:29.208767+0000","last_active":"2024-04-25T14:04:31.220710+0000","last_peered":"2024-04-25T14:04:31.220710+0000","last_clean":"2024-04-25T14:04:31.220710+0000","last_became_active":"2024-04-25T14:04:29.207051+0000","last_became_peered":"2024-04-25T14:04:29.207051+0000","last_unstale":"2024-04-25T14:04:31.220710+0000","last_undegraded":"2024-04-25T14:04:31.220710+0000","last_fullsized":"2024-04-25T14:04:31.220710+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:15.406391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229312+0000","last_change":"2024-04-25T14:04:30.218526+0000","last_active":"2024-04-25T14:04:30.229312+0000","last_peered":"2024-04-25T14:04:30.229312+0000","last_clean":"2024-04-25T14:04:30.229312+0000","last_became_active":"2024-04-25T14:04:30.218230+0000","last_became_peered":"2024-04-25T14:04:30.218230+0000","last_unstale":"2024-04-25T14:04:30.229312+0000","last_undegraded":"2024-04-25T14:04:30.229312+0000","last_fullsized":"2024-04-25T14:04:30.229312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.207563+0000","last_change":"2024-04-25T14:04:29.217799+0000","last_active":"2024-04-25T14:04:30.207563+0000","last_peered":"2024-04-25T14:04:30.207563+0000","last_clean":"2024-04-25T14:04:30.207563+0000","last_became_active":"2024-04-25T14:04:29.217428+0000","last_became_peered":"2024-04-25T14:04:29.217428+0000","last_unstale":"2024-04-25T14:04:30.207563+0000","last_undegraded":"2024-04-25T14:04:30.207563+0000","last_fullsized":"2024-04-25T14:04:30.207563+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:41.629406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225586+0000","last_change":"2024-04-25T14:04:30.220410+0000","last_active":"2024-04-25T14:04:30.225586+0000","last_peered":"2024-04-25T14:04:30.225586+0000","last_clean":"2024-04-25T14:04:30.225586+0000","last_became_active":"2024-04-25T14:04:30.220120+0000","last_became_peered":"2024-04-25T14:04:30.220120+0000","last_unstale":"2024-04-25T14:04:30.225586+0000","last_undegraded":"2024-04-25T14:04:30.225586+0000","last_fullsized":"2024-04-25T14:04:30.225586+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:33.189618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203311+0000","last_change":"2024-04-25T14:04:29.219179+0000","last_active":"2024-04-25T14:04:30.203311+0000","last_peered":"2024-04-25T14:04:30.203311+0000","last_clean":"2024-04-25T14:04:30.203311+0000","last_became_active":"2024-04-25T14:04:29.218839+0000","last_became_peered":"2024-04-25T14:04:29.218839+0000","last_unstale":"2024-04-25T14:04:30.203311+0000","last_undegraded":"2024-04-25T14:04:30.203311+0000","last_fullsized":"2024-04-25T14:04:30.203311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:12.979944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226284+0000","last_change":"2024-04-25T14:04:30.221377+0000","last_active":"2024-04-25T14:04:30.226284+0000","last_peered":"2024-04-25T14:04:30.226284+0000","last_clean":"2024-04-25T14:04:30.226284+0000","last_became_active":"2024-04-25T14:04:30.216730+0000","last_became_peered":"2024-04-25T14:04:30.216730+0000","last_unstale":"2024-04-25T14:04:30.226284+0000","last_undegraded":"2024-04-25T14:04:30.226284+0000","last_fullsized":"2024-04-25T14:04:30.226284+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:29.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.202738+0000","last_change":"2024-04-25T14:04:29.214185+0000","last_active":"2024-04-25T14:04:30.202738+0000","last_peered":"2024-04-25T14:04:30.202738+0000","last_clean":"2024-04-25T14:04:30.202738+0000","last_became_active":"2024-04-25T14:04:29.213861+0000","last_became_peered":"2024-04-25T14:04:29.213861+0000","last_unstale":"2024-04-25T14:04:30.202738+0000","last_undegraded":"2024-04-25T14:04:30.202738+0000","last_fullsized":"2024-04-25T14:04:30.202738+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:16.742155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231378+0000","last_change":"2024-04-25T14:04:30.226343+0000","last_active":"2024-04-25T14:04:30.231378+0000","last_peered":"2024-04-25T14:04:30.231378+0000","last_clean":"2024-04-25T14:04:30.231378+0000","last_became_active":"2024-04-25T14:04:30.225993+0000","last_became_peered":"2024-04-25T14:04:30.225993+0000","last_unstale":"2024-04-25T14:04:30.231378+0000","last_undegraded":"2024-04-25T14:04:30.231378+0000","last_fullsized":"2024-04-25T14:04:30.231378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220864+0000","last_change":"2024-04-25T14:04:29.210709+0000","last_active":"2024-04-25T14:04:31.220864+0000","last_peered":"2024-04-25T14:04:31.220864+0000","last_clean":"2024-04-25T14:04:31.220864+0000","last_became_active":"2024-04-25T14:04:29.210390+0000","last_became_peered":"2024-04-25T14:04:29.210390+0000","last_unstale":"2024-04-25T14:04:31.220864+0000","last_undegraded":"2024-04-25T14:04:31.220864+0000","last_fullsized":"2024-04-25T14:04:31.220864+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:11.326588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229262+0000","last_change":"2024-04-25T14:04:30.211440+0000","last_active":"2024-04-25T14:04:30.229262+0000","last_peered":"2024-04-25T14:04:30.229262+0000","last_clean":"2024-04-25T14:04:30.229262+0000","last_became_active":"2024-04-25T14:04:30.211139+0000","last_became_peered":"2024-04-25T14:04:30.211139+0000","last_unstale":"2024-04-25T14:04:30.229262+0000","last_undegraded":"2024-04-25T14:04:30.229262+0000","last_fullsized":"2024-04-25T14:04:30.229262+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.206648+0000","last_change":"2024-04-25T14:04:29.208231+0000","last_active":"2024-04-25T14:04:30.206648+0000","last_peered":"2024-04-25T14:04:30.206648+0000","last_clean":"2024-04-25T14:04:30.206648+0000","last_became_active":"2024-04-25T14:04:29.207908+0000","last_became_peered":"2024-04-25T14:04:29.207908+0000","last_unstale":"2024-04-25T14:04:30.206648+0000","last_undegraded":"2024-04-25T14:04:30.206648+0000","last_fullsized":"2024-04-25T14:04:30.206648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.915963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219919+0000","last_change":"2024-04-25T14:04:30.221325+0000","last_active":"2024-04-25T14:04:31.219919+0000","last_peered":"2024-04-25T14:04:31.219919+0000","last_clean":"2024-04-25T14:04:31.219919+0000","last_became_active":"2024-04-25T14:04:30.220805+0000","last_became_peered":"2024-04-25T14:04:30.220805+0000","last_unstale":"2024-04-25T14:04:31.219919+0000","last_undegraded":"2024-04-25T14:04:31.219919+0000","last_fullsized":"2024-04-25T14:04:31.219919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216996+0000","last_change":"2024-04-25T14:04:29.219631+0000","last_active":"2024-04-25T14:04:30.216996+0000","last_peered":"2024-04-25T14:04:30.216996+0000","last_clean":"2024-04-25T14:04:30.216996+0000","last_became_active":"2024-04-25T14:04:29.218130+0000","last_became_peered":"2024-04-25T14:04:29.218130+0000","last_unstale":"2024-04-25T14:04:30.216996+0000","last_undegraded":"2024-04-25T14:04:30.216996+0000","last_fullsized":"2024-04-25T14:04:30.216996+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:08.521825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":183,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220049+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:04:31.220049+0000","last_peered":"2024-04-25T14:04:31.220049+0000","last_clean":"2024-04-25T14:04:31.220049+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:04:31.220049+0000","last_undegraded":"2024-04-25T14:04:31.220049+0000","last_fullsized":"2024-04-25T14:04:31.220049+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1269760,"data_stored":894013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":304173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":296829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":659456,"data_stored":297411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":298463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":667648,"data_stored":298441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":296989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1257472,"data_stored":887253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:04:33.215 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:04:33.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.359+0000 7f271f772640 1 Processor -- start 2024-04-25T14:04:33.359 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.359+0000 7f271f772640 1 -- start start 2024-04-25T14:04:33.360 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f271815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2718070e70 0x7f271806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 0x7f2718160180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f271806d750 con 0x7f2718070e70 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f27180afe00 con 0x7f271806d340 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271f772640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27180b00a0 con 0x7f2718156cd0 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2718070e70 0x7f271806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f2718070e70 0x7f271806ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59956/0 (socket says 172.21.15.150:59956) 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 -- 172.21.15.150:0/4159235816 learned_addr learned my addr 172.21.15.150:0/4159235816 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271d50e640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f271815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271dd0f640 1 --2- 172.21.15.150:0/4159235816 >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 0x7f2718160180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 -- 172.21.15.150:0/4159235816 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 msgr2=0x7f271815d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f271815d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.361 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 -- 172.21.15.150:0/4159235816 >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 msgr2=0x7f2718160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:33.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 --2- 172.21.15.150:0/4159235816 >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 0x7f2718160180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271cd0d640 1 -- 172.21.15.150:0/4159235816 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27180b0380 con 0x7f2718070e70 2024-04-25T14:04:33.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271d50e640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f271815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:33.362 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.361+0000 7f271dd0f640 1 --2- 172.21.15.150:0/4159235816 >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 0x7f2718160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.362+0000 7f271cd0d640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f271806ce00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f271806e030 tx=0x7f270802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2195818d36bbaa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.362+0000 7f270ffff640 1 -- 172.21.15.150:0/4159235816 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708037070 con 0x7f2718070e70 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.362+0000 7f270ffff640 1 -- 172.21.15.150:0/4159235816 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2708032440 con 0x7f2718070e70 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.362+0000 7f270ffff640 1 -- 172.21.15.150:0/4159235816 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708032810 con 0x7f2718070e70 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 -- 172.21.15.150:0/4159235816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 msgr2=0x7f271806ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.363 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f271806ce00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f271806e030 tx=0x7f270802a1e0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 -- 172.21.15.150:0/4159235816 shutdown_connections 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 --2- 172.21.15.150:0/4159235816 >> v2:172.21.15.187:3300/0 conn(0x7f271806d340 0x7f2718160180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f271806ce00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 --2- 172.21.15.150:0/4159235816 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f271815d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 -- 172.21.15.150:0/4159235816 >> 172.21.15.150:0/4159235816 conn(0x7f27180d1ae0 msgr2=0x7f27180d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 -- 172.21.15.150:0/4159235816 shutdown_connections 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.363+0000 7f271f772640 1 -- 172.21.15.150:0/4159235816 wait complete. 2024-04-25T14:04:33.364 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.365+0000 7f271f772640 1 Processor -- start 2024-04-25T14:04:33.365 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.365+0000 7f271f772640 1 -- start start 2024-04-25T14:04:33.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f271806d340 0x7f2718149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f27181f51e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27180b0e60 con 0x7f2718070e70 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27180b0940 con 0x7f271806d340 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271f772640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f2718071280 con 0x7f2718156cd0 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271cd0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271cd0d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59966/0 (socket says 172.21.15.150:59966) 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271cd0d640 1 -- 172.21.15.150:0/3268778049 learned_addr learned my addr 172.21.15.150:0/3268778049 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271d50e640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f271806d340 0x7f2718149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.366+0000 7f271dd0f640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f27181f51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 -- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 msgr2=0x7f27181f51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f27181f51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.367 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 -- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f271806d340 msgr2=0x7f2718149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f271806d340 0x7f2718149080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 -- 172.21.15.150:0/3268778049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27181f59f0 con 0x7f2718070e70 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271cd0d640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2708000c00 tx=0x7f27080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708041020 con 0x7f2718070e70 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27080306c0 con 0x7f2718070e70 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708037040 con 0x7f2718070e70 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27181f81b0 con 0x7f2718070e70 2024-04-25T14:04:33.368 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.367+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27181f8770 con 0x7f2718070e70 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.368+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2718076a40 con 0x7f2718070e70 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.369+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2708044020 con 0x7f2718070e70 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.369+0000 7f270dffb640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 0x7f26fc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.369+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f27080b5b30 con 0x7f2718070e70 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.369+0000 7f271d50e640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 0x7f26fc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.370+0000 7f271d50e640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 0x7f26fc0769c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2718074640 tx=0x7f2714009870 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.374+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f270804b030 con 0x7f2718070e70 2024-04-25T14:04:33.544 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.545+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f27181f5780 con 0x7f26fc0744b0 2024-04-25T14:04:33.550 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.551+0000 7f270dffb640 1 -- 172.21.15.150:0/3268778049 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251898 (secure 0 0 0) 0x7f27181f5780 con 0x7f26fc0744b0 2024-04-25T14:04:33.553 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:33.554 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 msgr2=0x7f26fc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 0x7f26fc0769c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f2718074640 tx=0x7f2714009870 comp rx=0 tx=0).stop 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 msgr2=0x7f27181f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f2708000c00 tx=0x7f27080046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 shutdown_connections 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f26fc0744b0 0x7f26fc0769c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2718156cd0 0x7f27181f51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.556 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f2718070e70 0x7f27181f4ca0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 --2- 172.21.15.150:0/3268778049 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f271806d340 0x7f2718149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 >> 172.21.15.150:0/3268778049 conn(0x7f27180d1ae0 msgr2=0x7f271815d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 shutdown_connections 2024-04-25T14:04:33.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.554+0000 7f271f772640 1 -- 172.21.15.150:0/3268778049 wait complete. 2024-04-25T14:04:33.565 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-04-25T14:04:32.619397+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222544,"kb_used_data":6368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752176,"statfs":{"total":773094113280,"available":772866228224,"internally_reserved":0,"allocated":6520832,"data_stored":3573572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.431389"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.218963+0000","last_change":"2024-04-25T14:04:30.214547+0000","last_active":"2024-04-25T14:04:30.218963+0000","last_peered":"2024-04-25T14:04:30.218963+0000","last_clean":"2024-04-25T14:04:30.218963+0000","last_became_active":"2024-04-25T14:04:30.214243+0000","last_became_peered":"2024-04-25T14:04:30.214243+0000","last_unstale":"2024-04-25T14:04:30.218963+0000","last_undegraded":"2024-04-25T14:04:30.218963+0000","last_fullsized":"2024-04-25T14:04:30.218963+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222180+0000","last_change":"2024-04-25T14:04:30.219227+0000","last_active":"2024-04-25T14:04:31.222180+0000","last_peered":"2024-04-25T14:04:31.222180+0000","last_clean":"2024-04-25T14:04:31.222180+0000","last_became_active":"2024-04-25T14:04:30.218875+0000","last_became_peered":"2024-04-25T14:04:30.218875+0000","last_unstale":"2024-04-25T14:04:31.222180+0000","last_undegraded":"2024-04-25T14:04:31.222180+0000","last_fullsized":"2024-04-25T14:04:31.222180+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231177+0000","last_change":"2024-04-25T14:04:30.226045+0000","last_active":"2024-04-25T14:04:30.231177+0000","last_peered":"2024-04-25T14:04:30.231177+0000","last_clean":"2024-04-25T14:04:30.231177+0000","last_became_active":"2024-04-25T14:04:30.225740+0000","last_became_peered":"2024-04-25T14:04:30.225740+0000","last_unstale":"2024-04-25T14:04:30.231177+0000","last_undegraded":"2024-04-25T14:04:30.231177+0000","last_fullsized":"2024-04-25T14:04:30.231177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222720+0000","last_change":"2024-04-25T14:04:30.214708+0000","last_active":"2024-04-25T14:04:31.222720+0000","last_peered":"2024-04-25T14:04:31.222720+0000","last_clean":"2024-04-25T14:04:31.222720+0000","last_became_active":"2024-04-25T14:04:30.214430+0000","last_became_peered":"2024-04-25T14:04:30.214430+0000","last_unstale":"2024-04-25T14:04:31.222720+0000","last_undegraded":"2024-04-25T14:04:31.222720+0000","last_fullsized":"2024-04-25T14:04:31.222720+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221187+0000","last_change":"2024-04-25T14:04:30.215762+0000","last_active":"2024-04-25T14:04:31.221187+0000","last_peered":"2024-04-25T14:04:31.221187+0000","last_clean":"2024-04-25T14:04:31.221187+0000","last_became_active":"2024-04-25T14:04:30.215432+0000","last_became_peered":"2024-04-25T14:04:30.215432+0000","last_unstale":"2024-04-25T14:04:31.221187+0000","last_undegraded":"2024-04-25T14:04:31.221187+0000","last_fullsized":"2024-04-25T14:04:31.221187+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215995+0000","last_change":"2024-04-25T14:04:30.211240+0000","last_active":"2024-04-25T14:04:30.215995+0000","last_peered":"2024-04-25T14:04:30.215995+0000","last_clean":"2024-04-25T14:04:30.215995+0000","last_became_active":"2024-04-25T14:04:30.210897+0000","last_became_peered":"2024-04-25T14:04:30.210897+0000","last_unstale":"2024-04-25T14:04:30.215995+0000","last_undegraded":"2024-04-25T14:04:30.215995+0000","last_fullsized":"2024-04-25T14:04:30.215995+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:18:45.625324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231104+0000","last_change":"2024-04-25T14:04:30.226241+0000","last_active":"2024-04-25T14:04:30.231104+0000","last_peered":"2024-04-25T14:04:30.231104+0000","last_clean":"2024-04-25T14:04:30.231104+0000","last_became_active":"2024-04-25T14:04:30.225929+0000","last_became_peered":"2024-04-25T14:04:30.225929+0000","last_unstale":"2024-04-25T14:04:30.231104+0000","last_undegraded":"2024-04-25T14:04:30.231104+0000","last_fullsized":"2024-04-25T14:04:30.231104+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229563+0000","last_change":"2024-04-25T14:04:30.223557+0000","last_active":"2024-04-25T14:04:30.229563+0000","last_peered":"2024-04-25T14:04:30.229563+0000","last_clean":"2024-04-25T14:04:30.229563+0000","last_became_active":"2024-04-25T14:04:30.221176+0000","last_became_peered":"2024-04-25T14:04:30.221176+0000","last_unstale":"2024-04-25T14:04:30.229563+0000","last_undegraded":"2024-04-25T14:04:30.229563+0000","last_fullsized":"2024-04-25T14:04:30.229563+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:35.801013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229299+0000","last_change":"2024-04-25T14:04:30.221045+0000","last_active":"2024-04-25T14:04:30.229299+0000","last_peered":"2024-04-25T14:04:30.229299+0000","last_clean":"2024-04-25T14:04:30.229299+0000","last_became_active":"2024-04-25T14:04:30.220661+0000","last_became_peered":"2024-04-25T14:04:30.220661+0000","last_unstale":"2024-04-25T14:04:30.229299+0000","last_undegraded":"2024-04-25T14:04:30.229299+0000","last_fullsized":"2024-04-25T14:04:30.229299+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.205161+0000","last_change":"2024-04-25T14:04:30.203891+0000","last_active":"2024-04-25T14:04:30.205161+0000","last_peered":"2024-04-25T14:04:30.205161+0000","last_clean":"2024-04-25T14:04:30.205161+0000","last_became_active":"2024-04-25T14:04:30.203430+0000","last_became_peered":"2024-04-25T14:04:30.203430+0000","last_unstale":"2024-04-25T14:04:30.205161+0000","last_undegraded":"2024-04-25T14:04:30.205161+0000","last_fullsized":"2024-04-25T14:04:30.205161+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:12:48.736540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215641+0000","last_change":"2024-04-25T14:04:30.211972+0000","last_active":"2024-04-25T14:04:30.215641+0000","last_peered":"2024-04-25T14:04:30.215641+0000","last_clean":"2024-04-25T14:04:30.215641+0000","last_became_active":"2024-04-25T14:04:30.211695+0000","last_became_peered":"2024-04-25T14:04:30.211695+0000","last_unstale":"2024-04-25T14:04:30.215641+0000","last_undegraded":"2024-04-25T14:04:30.215641+0000","last_fullsized":"2024-04-25T14:04:30.215641+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:08:33.449981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231239+0000","last_change":"2024-04-25T14:04:30.227914+0000","last_active":"2024-04-25T14:04:30.231239+0000","last_peered":"2024-04-25T14:04:30.231239+0000","last_clean":"2024-04-25T14:04:30.231239+0000","last_became_active":"2024-04-25T14:04:30.227403+0000","last_became_peered":"2024-04-25T14:04:30.227403+0000","last_unstale":"2024-04-25T14:04:30.231239+0000","last_undegraded":"2024-04-25T14:04:30.231239+0000","last_fullsized":"2024-04-25T14:04:30.231239+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:02:04.752951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.205408+0000","last_change":"2024-04-25T14:04:30.204277+0000","last_active":"2024-04-25T14:04:30.205408+0000","last_peered":"2024-04-25T14:04:30.205408+0000","last_clean":"2024-04-25T14:04:30.205408+0000","last_became_active":"2024-04-25T14:04:30.203881+0000","last_became_peered":"2024-04-25T14:04:30.203881+0000","last_unstale":"2024-04-25T14:04:30.205408+0000","last_undegraded":"2024-04-25T14:04:30.205408+0000","last_fullsized":"2024-04-25T14:04:30.205408+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:39.227710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226642+0000","last_change":"2024-04-25T14:04:30.224171+0000","last_active":"2024-04-25T14:04:30.226642+0000","last_peered":"2024-04-25T14:04:30.226642+0000","last_clean":"2024-04-25T14:04:30.226642+0000","last_became_active":"2024-04-25T14:04:30.223883+0000","last_became_peered":"2024-04-25T14:04:30.223883+0000","last_unstale":"2024-04-25T14:04:30.226642+0000","last_undegraded":"2024-04-25T14:04:30.226642+0000","last_fullsized":"2024-04-25T14:04:30.226642+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221836+0000","last_change":"2024-04-25T14:04:30.220916+0000","last_active":"2024-04-25T14:04:31.221836+0000","last_peered":"2024-04-25T14:04:31.221836+0000","last_clean":"2024-04-25T14:04:31.221836+0000","last_became_active":"2024-04-25T14:04:30.220620+0000","last_became_peered":"2024-04-25T14:04:30.220620+0000","last_unstale":"2024-04-25T14:04:31.221836+0000","last_undegraded":"2024-04-25T14:04:31.221836+0000","last_fullsized":"2024-04-25T14:04:31.221836+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.214587+0000","last_change":"2024-04-25T14:04:30.205816+0000","last_active":"2024-04-25T14:04:30.214587+0000","last_peered":"2024-04-25T14:04:30.214587+0000","last_clean":"2024-04-25T14:04:30.214587+0000","last_became_active":"2024-04-25T14:04:30.205453+0000","last_became_peered":"2024-04-25T14:04:30.205453+0000","last_unstale":"2024-04-25T14:04:30.214587+0000","last_undegraded":"2024-04-25T14:04:30.214587+0000","last_fullsized":"2024-04-25T14:04:30.214587+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:50:02.742205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224044+0000","last_change":"2024-04-25T14:04:30.218132+0000","last_active":"2024-04-25T14:04:30.224044+0000","last_peered":"2024-04-25T14:04:30.224044+0000","last_clean":"2024-04-25T14:04:30.224044+0000","last_became_active":"2024-04-25T14:04:30.217293+0000","last_became_peered":"2024-04-25T14:04:30.217293+0000","last_unstale":"2024-04-25T14:04:30.224044+0000","last_undegraded":"2024-04-25T14:04:30.224044+0000","last_fullsized":"2024-04-25T14:04:30.224044+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:40.569444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222395+0000","last_change":"2024-04-25T14:04:30.218172+0000","last_active":"2024-04-25T14:04:31.222395+0000","last_peered":"2024-04-25T14:04:31.222395+0000","last_clean":"2024-04-25T14:04:31.222395+0000","last_became_active":"2024-04-25T14:04:30.217788+0000","last_became_peered":"2024-04-25T14:04:30.217788+0000","last_unstale":"2024-04-25T14:04:31.222395+0000","last_undegraded":"2024-04-25T14:04:31.222395+0000","last_fullsized":"2024-04-25T14:04:31.222395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.227288+0000","last_change":"2024-04-25T14:04:30.223862+0000","last_active":"2024-04-25T14:04:30.227288+0000","last_peered":"2024-04-25T14:04:30.227288+0000","last_clean":"2024-04-25T14:04:30.227288+0000","last_became_active":"2024-04-25T14:04:30.223569+0000","last_became_peered":"2024-04-25T14:04:30.223569+0000","last_unstale":"2024-04-25T14:04:30.227288+0000","last_undegraded":"2024-04-25T14:04:30.227288+0000","last_fullsized":"2024-04-25T14:04:30.227288+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219951+0000","last_change":"2024-04-25T14:04:30.220878+0000","last_active":"2024-04-25T14:04:31.219951+0000","last_peered":"2024-04-25T14:04:31.219951+0000","last_clean":"2024-04-25T14:04:31.219951+0000","last_became_active":"2024-04-25T14:04:30.220575+0000","last_became_peered":"2024-04-25T14:04:30.220575+0000","last_unstale":"2024-04-25T14:04:31.219951+0000","last_undegraded":"2024-04-25T14:04:31.219951+0000","last_fullsized":"2024-04-25T14:04:31.219951+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220438+0000","last_change":"2024-04-25T14:04:30.224581+0000","last_active":"2024-04-25T14:04:31.220438+0000","last_peered":"2024-04-25T14:04:31.220438+0000","last_clean":"2024-04-25T14:04:31.220438+0000","last_became_active":"2024-04-25T14:04:30.224277+0000","last_became_peered":"2024-04-25T14:04:30.224277+0000","last_unstale":"2024-04-25T14:04:31.220438+0000","last_undegraded":"2024-04-25T14:04:31.220438+0000","last_fullsized":"2024-04-25T14:04:31.220438+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:10.900532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231225+0000","last_change":"2024-04-25T14:04:30.228708+0000","last_active":"2024-04-25T14:04:30.231225+0000","last_peered":"2024-04-25T14:04:30.231225+0000","last_clean":"2024-04-25T14:04:30.231225+0000","last_became_active":"2024-04-25T14:04:30.228416+0000","last_became_peered":"2024-04-25T14:04:30.228416+0000","last_unstale":"2024-04-25T14:04:30.231225+0000","last_undegraded":"2024-04-25T14:04:30.231225+0000","last_fullsized":"2024-04-25T14:04:30.231225+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:48:42.490744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222467+0000","last_change":"2024-04-25T14:04:30.215506+0000","last_active":"2024-04-25T14:04:30.222467+0000","last_peered":"2024-04-25T14:04:30.222467+0000","last_clean":"2024-04-25T14:04:30.222467+0000","last_became_active":"2024-04-25T14:04:30.215222+0000","last_became_peered":"2024-04-25T14:04:30.215222+0000","last_unstale":"2024-04-25T14:04:30.222467+0000","last_undegraded":"2024-04-25T14:04:30.222467+0000","last_fullsized":"2024-04-25T14:04:30.222467+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:44:02.964930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225126+0000","last_change":"2024-04-25T14:04:30.218465+0000","last_active":"2024-04-25T14:04:30.225126+0000","last_peered":"2024-04-25T14:04:30.225126+0000","last_clean":"2024-04-25T14:04:30.225126+0000","last_became_active":"2024-04-25T14:04:30.218179+0000","last_became_peered":"2024-04-25T14:04:30.218179+0000","last_unstale":"2024-04-25T14:04:30.225126+0000","last_undegraded":"2024-04-25T14:04:30.225126+0000","last_fullsized":"2024-04-25T14:04:30.225126+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.794183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225327+0000","last_change":"2024-04-25T14:04:30.222735+0000","last_active":"2024-04-25T14:04:30.225327+0000","last_peered":"2024-04-25T14:04:30.225327+0000","last_clean":"2024-04-25T14:04:30.225327+0000","last_became_active":"2024-04-25T14:04:30.221730+0000","last_became_peered":"2024-04-25T14:04:30.221730+0000","last_unstale":"2024-04-25T14:04:30.225327+0000","last_undegraded":"2024-04-25T14:04:30.225327+0000","last_fullsized":"2024-04-25T14:04:30.225327+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:13.981930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216593+0000","last_change":"2024-04-25T14:04:30.210180+0000","last_active":"2024-04-25T14:04:30.216593+0000","last_peered":"2024-04-25T14:04:30.216593+0000","last_clean":"2024-04-25T14:04:30.216593+0000","last_became_active":"2024-04-25T14:04:30.209890+0000","last_became_peered":"2024-04-25T14:04:30.209890+0000","last_unstale":"2024-04-25T14:04:30.216593+0000","last_undegraded":"2024-04-25T14:04:30.216593+0000","last_fullsized":"2024-04-25T14:04:30.216593+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231177+0000","last_change":"2024-04-25T14:04:30.227883+0000","last_active":"2024-04-25T14:04:30.231177+0000","last_peered":"2024-04-25T14:04:30.231177+0000","last_clean":"2024-04-25T14:04:30.231177+0000","last_became_active":"2024-04-25T14:04:30.227598+0000","last_became_peered":"2024-04-25T14:04:30.227598+0000","last_unstale":"2024-04-25T14:04:30.231177+0000","last_undegraded":"2024-04-25T14:04:30.231177+0000","last_fullsized":"2024-04-25T14:04:30.231177+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:10.900532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220172+0000","last_change":"2024-04-25T14:04:30.218772+0000","last_active":"2024-04-25T14:04:31.220172+0000","last_peered":"2024-04-25T14:04:31.220172+0000","last_clean":"2024-04-25T14:04:31.220172+0000","last_became_active":"2024-04-25T14:04:30.218475+0000","last_became_peered":"2024-04-25T14:04:30.218475+0000","last_unstale":"2024-04-25T14:04:31.220172+0000","last_undegraded":"2024-04-25T14:04:31.220172+0000","last_fullsized":"2024-04-25T14:04:31.220172+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:45.046031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231835+0000","last_change":"2024-04-25T14:04:30.230664+0000","last_active":"2024-04-25T14:04:30.231835+0000","last_peered":"2024-04-25T14:04:30.231835+0000","last_clean":"2024-04-25T14:04:30.231835+0000","last_became_active":"2024-04-25T14:04:30.229801+0000","last_became_peered":"2024-04-25T14:04:30.229801+0000","last_unstale":"2024-04-25T14:04:30.231835+0000","last_undegraded":"2024-04-25T14:04:30.231835+0000","last_fullsized":"2024-04-25T14:04:30.231835+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:35.801013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.222395+0000","last_change":"2024-04-25T14:04:30.214418+0000","last_active":"2024-04-25T14:04:31.222395+0000","last_peered":"2024-04-25T14:04:31.222395+0000","last_clean":"2024-04-25T14:04:31.222395+0000","last_became_active":"2024-04-25T14:04:30.214126+0000","last_became_peered":"2024-04-25T14:04:30.214126+0000","last_unstale":"2024-04-25T14:04:31.222395+0000","last_undegraded":"2024-04-25T14:04:31.222395+0000","last_fullsized":"2024-04-25T14:04:31.222395+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226671+0000","last_change":"2024-04-25T14:04:30.223706+0000","last_active":"2024-04-25T14:04:30.226671+0000","last_peered":"2024-04-25T14:04:30.226671+0000","last_clean":"2024-04-25T14:04:30.226671+0000","last_became_active":"2024-04-25T14:04:30.223403+0000","last_became_peered":"2024-04-25T14:04:30.223403+0000","last_unstale":"2024-04-25T14:04:30.226671+0000","last_undegraded":"2024-04-25T14:04:30.226671+0000","last_fullsized":"2024-04-25T14:04:30.226671+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226144+0000","last_change":"2024-04-25T14:04:30.222373+0000","last_active":"2024-04-25T14:04:30.226144+0000","last_peered":"2024-04-25T14:04:30.226144+0000","last_clean":"2024-04-25T14:04:30.226144+0000","last_became_active":"2024-04-25T14:04:30.221976+0000","last_became_peered":"2024-04-25T14:04:30.221976+0000","last_unstale":"2024-04-25T14:04:30.226144+0000","last_undegraded":"2024-04-25T14:04:30.226144+0000","last_fullsized":"2024-04-25T14:04:30.226144+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:13.981930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219925+0000","last_change":"2024-04-25T14:04:30.218727+0000","last_active":"2024-04-25T14:04:31.219925+0000","last_peered":"2024-04-25T14:04:31.219925+0000","last_clean":"2024-04-25T14:04:31.219925+0000","last_became_active":"2024-04-25T14:04:30.218414+0000","last_became_peered":"2024-04-25T14:04:30.218414+0000","last_unstale":"2024-04-25T14:04:31.219925+0000","last_undegraded":"2024-04-25T14:04:31.219925+0000","last_fullsized":"2024-04-25T14:04:31.219925+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"55'4","reported_seq":18,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220565+0000","last_change":"2024-04-25T14:04:29.212267+0000","last_active":"2024-04-25T14:04:31.220565+0000","last_peered":"2024-04-25T14:04:31.220565+0000","last_clean":"2024-04-25T14:04:31.220565+0000","last_became_active":"2024-04-25T14:04:29.211937+0000","last_became_peered":"2024-04-25T14:04:29.211937+0000","last_unstale":"2024-04-25T14:04:31.220565+0000","last_undegraded":"2024-04-25T14:04:31.220565+0000","last_fullsized":"2024-04-25T14:04:31.220565+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:53:23.626937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231252+0000","last_change":"2024-04-25T14:04:30.223856+0000","last_active":"2024-04-25T14:04:30.231252+0000","last_peered":"2024-04-25T14:04:30.231252+0000","last_clean":"2024-04-25T14:04:30.231252+0000","last_became_active":"2024-04-25T14:04:30.223557+0000","last_became_peered":"2024-04-25T14:04:30.223557+0000","last_unstale":"2024-04-25T14:04:30.231252+0000","last_undegraded":"2024-04-25T14:04:30.231252+0000","last_fullsized":"2024-04-25T14:04:30.231252+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229955+0000","last_change":"2024-04-25T14:04:29.217418+0000","last_active":"2024-04-25T14:04:30.229955+0000","last_peered":"2024-04-25T14:04:30.229955+0000","last_clean":"2024-04-25T14:04:30.229955+0000","last_became_active":"2024-04-25T14:04:29.216344+0000","last_became_peered":"2024-04-25T14:04:29.216344+0000","last_unstale":"2024-04-25T14:04:30.229955+0000","last_undegraded":"2024-04-25T14:04:30.229955+0000","last_fullsized":"2024-04-25T14:04:30.229955+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:42:20.310590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221300+0000","last_change":"2024-04-25T14:04:30.223981+0000","last_active":"2024-04-25T14:04:31.221300+0000","last_peered":"2024-04-25T14:04:31.221300+0000","last_clean":"2024-04-25T14:04:31.221300+0000","last_became_active":"2024-04-25T14:04:30.223463+0000","last_became_peered":"2024-04-25T14:04:30.223463+0000","last_unstale":"2024-04-25T14:04:31.221300+0000","last_undegraded":"2024-04-25T14:04:31.221300+0000","last_fullsized":"2024-04-25T14:04:31.221300+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:16.408377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224876+0000","last_change":"2024-04-25T14:04:29.211979+0000","last_active":"2024-04-25T14:04:30.224876+0000","last_peered":"2024-04-25T14:04:30.224876+0000","last_clean":"2024-04-25T14:04:30.224876+0000","last_became_active":"2024-04-25T14:04:29.211471+0000","last_became_peered":"2024-04-25T14:04:29.211471+0000","last_unstale":"2024-04-25T14:04:30.224876+0000","last_undegraded":"2024-04-25T14:04:30.224876+0000","last_fullsized":"2024-04-25T14:04:30.224876+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:49:01.076515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224979+0000","last_change":"2024-04-25T14:04:30.209984+0000","last_active":"2024-04-25T14:04:30.224979+0000","last_peered":"2024-04-25T14:04:30.224979+0000","last_clean":"2024-04-25T14:04:30.224979+0000","last_became_active":"2024-04-25T14:04:30.209645+0000","last_became_peered":"2024-04-25T14:04:30.209645+0000","last_unstale":"2024-04-25T14:04:30.224979+0000","last_undegraded":"2024-04-25T14:04:30.224979+0000","last_fullsized":"2024-04-25T14:04:30.224979+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231441+0000","last_change":"2024-04-25T14:04:29.213802+0000","last_active":"2024-04-25T14:04:30.231441+0000","last_peered":"2024-04-25T14:04:30.231441+0000","last_clean":"2024-04-25T14:04:30.231441+0000","last_became_active":"2024-04-25T14:04:29.213479+0000","last_became_peered":"2024-04-25T14:04:29.213479+0000","last_unstale":"2024-04-25T14:04:30.231441+0000","last_undegraded":"2024-04-25T14:04:30.231441+0000","last_fullsized":"2024-04-25T14:04:30.231441+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:16:39.567458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221328+0000","last_change":"2024-04-25T14:04:30.217870+0000","last_active":"2024-04-25T14:04:31.221328+0000","last_peered":"2024-04-25T14:04:31.221328+0000","last_clean":"2024-04-25T14:04:31.221328+0000","last_became_active":"2024-04-25T14:04:30.217577+0000","last_became_peered":"2024-04-25T14:04:30.217577+0000","last_unstale":"2024-04-25T14:04:31.221328+0000","last_undegraded":"2024-04-25T14:04:31.221328+0000","last_fullsized":"2024-04-25T14:04:31.221328+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:32:06.638496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225103+0000","last_change":"2024-04-25T14:04:29.221871+0000","last_active":"2024-04-25T14:04:30.225103+0000","last_peered":"2024-04-25T14:04:30.225103+0000","last_clean":"2024-04-25T14:04:30.225103+0000","last_became_active":"2024-04-25T14:04:29.221524+0000","last_became_peered":"2024-04-25T14:04:29.221524+0000","last_unstale":"2024-04-25T14:04:30.225103+0000","last_undegraded":"2024-04-25T14:04:30.225103+0000","last_fullsized":"2024-04-25T14:04:30.225103+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:15.406391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220373+0000","last_change":"2024-04-25T14:04:30.218754+0000","last_active":"2024-04-25T14:04:31.220373+0000","last_peered":"2024-04-25T14:04:31.220373+0000","last_clean":"2024-04-25T14:04:31.220373+0000","last_became_active":"2024-04-25T14:04:30.218448+0000","last_became_peered":"2024-04-25T14:04:30.218448+0000","last_unstale":"2024-04-25T14:04:31.220373+0000","last_undegraded":"2024-04-25T14:04:31.220373+0000","last_fullsized":"2024-04-25T14:04:31.220373+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:05:28.550466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221434+0000","last_change":"2024-04-25T14:04:29.212086+0000","last_active":"2024-04-25T14:04:31.221434+0000","last_peered":"2024-04-25T14:04:31.221434+0000","last_clean":"2024-04-25T14:04:31.221434+0000","last_became_active":"2024-04-25T14:04:29.211254+0000","last_became_peered":"2024-04-25T14:04:29.211254+0000","last_unstale":"2024-04-25T14:04:31.221434+0000","last_undegraded":"2024-04-25T14:04:31.221434+0000","last_fullsized":"2024-04-25T14:04:31.221434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216620+0000","last_change":"2024-04-25T14:04:30.210445+0000","last_active":"2024-04-25T14:04:30.216620+0000","last_peered":"2024-04-25T14:04:30.216620+0000","last_clean":"2024-04-25T14:04:30.216620+0000","last_became_active":"2024-04-25T14:04:30.209874+0000","last_became_peered":"2024-04-25T14:04:30.209874+0000","last_unstale":"2024-04-25T14:04:30.216620+0000","last_undegraded":"2024-04-25T14:04:30.216620+0000","last_fullsized":"2024-04-25T14:04:30.216620+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220685+0000","last_change":"2024-04-25T14:04:29.208335+0000","last_active":"2024-04-25T14:04:31.220685+0000","last_peered":"2024-04-25T14:04:31.220685+0000","last_clean":"2024-04-25T14:04:31.220685+0000","last_became_active":"2024-04-25T14:04:29.207928+0000","last_became_peered":"2024-04-25T14:04:29.207928+0000","last_unstale":"2024-04-25T14:04:31.220685+0000","last_undegraded":"2024-04-25T14:04:31.220685+0000","last_fullsized":"2024-04-25T14:04:31.220685+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:29:49.048547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220355+0000","last_change":"2024-04-25T14:04:30.222225+0000","last_active":"2024-04-25T14:04:31.220355+0000","last_peered":"2024-04-25T14:04:31.220355+0000","last_clean":"2024-04-25T14:04:31.220355+0000","last_became_active":"2024-04-25T14:04:30.221901+0000","last_became_peered":"2024-04-25T14:04:30.221901+0000","last_unstale":"2024-04-25T14:04:31.220355+0000","last_undegraded":"2024-04-25T14:04:31.220355+0000","last_fullsized":"2024-04-25T14:04:31.220355+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:58:45.255285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.208221+0000","last_change":"2024-04-25T14:04:29.217227+0000","last_active":"2024-04-25T14:04:30.208221+0000","last_peered":"2024-04-25T14:04:30.208221+0000","last_clean":"2024-04-25T14:04:30.208221+0000","last_became_active":"2024-04-25T14:04:29.216904+0000","last_became_peered":"2024-04-25T14:04:29.216904+0000","last_unstale":"2024-04-25T14:04:30.208221+0000","last_undegraded":"2024-04-25T14:04:30.208221+0000","last_fullsized":"2024-04-25T14:04:30.208221+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:44.044045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220879+0000","last_change":"2024-04-25T14:04:30.219118+0000","last_active":"2024-04-25T14:04:31.220879+0000","last_peered":"2024-04-25T14:04:31.220879+0000","last_clean":"2024-04-25T14:04:31.220879+0000","last_became_active":"2024-04-25T14:04:30.214059+0000","last_became_peered":"2024-04-25T14:04:30.214059+0000","last_unstale":"2024-04-25T14:04:31.220879+0000","last_undegraded":"2024-04-25T14:04:31.220879+0000","last_fullsized":"2024-04-25T14:04:31.220879+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:25.133215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220277+0000","last_change":"2024-04-25T14:04:29.209016+0000","last_active":"2024-04-25T14:04:31.220277+0000","last_peered":"2024-04-25T14:04:31.220277+0000","last_clean":"2024-04-25T14:04:31.220277+0000","last_became_active":"2024-04-25T14:04:29.208687+0000","last_became_peered":"2024-04-25T14:04:29.208687+0000","last_unstale":"2024-04-25T14:04:31.220277+0000","last_undegraded":"2024-04-25T14:04:31.220277+0000","last_fullsized":"2024-04-25T14:04:31.220277+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:00.939721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.218186+0000","last_change":"2024-04-25T14:04:30.216937+0000","last_active":"2024-04-25T14:04:30.218186+0000","last_peered":"2024-04-25T14:04:30.218186+0000","last_clean":"2024-04-25T14:04:30.218186+0000","last_became_active":"2024-04-25T14:04:30.216609+0000","last_became_peered":"2024-04-25T14:04:30.216609+0000","last_unstale":"2024-04-25T14:04:30.218186+0000","last_undegraded":"2024-04-25T14:04:30.218186+0000","last_fullsized":"2024-04-25T14:04:30.218186+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:17.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221343+0000","last_change":"2024-04-25T14:04:29.221582+0000","last_active":"2024-04-25T14:04:31.221343+0000","last_peered":"2024-04-25T14:04:31.221343+0000","last_clean":"2024-04-25T14:04:31.221343+0000","last_became_active":"2024-04-25T14:04:29.221337+0000","last_became_peered":"2024-04-25T14:04:29.221337+0000","last_unstale":"2024-04-25T14:04:31.221343+0000","last_undegraded":"2024-04-25T14:04:31.221343+0000","last_fullsized":"2024-04-25T14:04:31.221343+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:51.800914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231767+0000","last_change":"2024-04-25T14:04:30.229612+0000","last_active":"2024-04-25T14:04:30.231767+0000","last_peered":"2024-04-25T14:04:30.231767+0000","last_clean":"2024-04-25T14:04:30.231767+0000","last_became_active":"2024-04-25T14:04:30.229322+0000","last_became_peered":"2024-04-25T14:04:30.229322+0000","last_unstale":"2024-04-25T14:04:30.231767+0000","last_undegraded":"2024-04-25T14:04:30.231767+0000","last_fullsized":"2024-04-25T14:04:30.231767+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"55'3","reported_seq":17,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220592+0000","last_change":"2024-04-25T14:04:29.211807+0000","last_active":"2024-04-25T14:04:31.220592+0000","last_peered":"2024-04-25T14:04:31.220592+0000","last_clean":"2024-04-25T14:04:31.220592+0000","last_became_active":"2024-04-25T14:04:29.211462+0000","last_became_peered":"2024-04-25T14:04:29.211462+0000","last_unstale":"2024-04-25T14:04:31.220592+0000","last_undegraded":"2024-04-25T14:04:31.220592+0000","last_fullsized":"2024-04-25T14:04:31.220592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:09.898546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221449+0000","last_change":"2024-04-25T14:04:30.207374+0000","last_active":"2024-04-25T14:04:31.221449+0000","last_peered":"2024-04-25T14:04:31.221449+0000","last_clean":"2024-04-25T14:04:31.221449+0000","last_became_active":"2024-04-25T14:04:30.207025+0000","last_became_peered":"2024-04-25T14:04:30.207025+0000","last_unstale":"2024-04-25T14:04:31.221449+0000","last_undegraded":"2024-04-25T14:04:31.221449+0000","last_fullsized":"2024-04-25T14:04:31.221449+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"55'6","reported_seq":18,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.234034+0000","last_change":"2024-04-25T14:04:29.219922+0000","last_active":"2024-04-25T14:04:30.234034+0000","last_peered":"2024-04-25T14:04:30.234034+0000","last_clean":"2024-04-25T14:04:30.234034+0000","last_became_active":"2024-04-25T14:04:29.219622+0000","last_became_peered":"2024-04-25T14:04:29.219622+0000","last_unstale":"2024-04-25T14:04:30.234034+0000","last_undegraded":"2024-04-25T14:04:30.234034+0000","last_fullsized":"2024-04-25T14:04:30.234034+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:32.187632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.220041+0000","last_change":"2024-04-25T14:04:30.217376+0000","last_active":"2024-04-25T14:04:30.220041+0000","last_peered":"2024-04-25T14:04:30.220041+0000","last_clean":"2024-04-25T14:04:30.220041+0000","last_became_active":"2024-04-25T14:04:30.217039+0000","last_became_peered":"2024-04-25T14:04:30.217039+0000","last_unstale":"2024-04-25T14:04:30.220041+0000","last_undegraded":"2024-04-25T14:04:30.220041+0000","last_fullsized":"2024-04-25T14:04:30.220041+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:33.189618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225214+0000","last_change":"2024-04-25T14:04:29.214433+0000","last_active":"2024-04-25T14:04:30.225214+0000","last_peered":"2024-04-25T14:04:30.225214+0000","last_clean":"2024-04-25T14:04:30.225214+0000","last_became_active":"2024-04-25T14:04:29.214086+0000","last_became_peered":"2024-04-25T14:04:29.214086+0000","last_unstale":"2024-04-25T14:04:30.225214+0000","last_undegraded":"2024-04-25T14:04:30.225214+0000","last_fullsized":"2024-04-25T14:04:30.225214+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:41:00.939721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224074+0000","last_change":"2024-04-25T14:04:30.218458+0000","last_active":"2024-04-25T14:04:30.224074+0000","last_peered":"2024-04-25T14:04:30.224074+0000","last_clean":"2024-04-25T14:04:30.224074+0000","last_became_active":"2024-04-25T14:04:30.218145+0000","last_became_peered":"2024-04-25T14:04:30.218145+0000","last_unstale":"2024-04-25T14:04:30.224074+0000","last_undegraded":"2024-04-25T14:04:30.224074+0000","last_fullsized":"2024-04-25T14:04:30.224074+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:37.917949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221060+0000","last_change":"2024-04-25T14:04:29.222899+0000","last_active":"2024-04-25T14:04:31.221060+0000","last_peered":"2024-04-25T14:04:31.221060+0000","last_clean":"2024-04-25T14:04:31.221060+0000","last_became_active":"2024-04-25T14:04:29.222626+0000","last_became_peered":"2024-04-25T14:04:29.222626+0000","last_unstale":"2024-04-25T14:04:31.221060+0000","last_undegraded":"2024-04-25T14:04:31.221060+0000","last_fullsized":"2024-04-25T14:04:31.221060+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:11.326588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229744+0000","last_change":"2024-04-25T14:04:30.219657+0000","last_active":"2024-04-25T14:04:30.229744+0000","last_peered":"2024-04-25T14:04:30.229744+0000","last_clean":"2024-04-25T14:04:30.229744+0000","last_became_active":"2024-04-25T14:04:30.219241+0000","last_became_peered":"2024-04-25T14:04:30.219241+0000","last_unstale":"2024-04-25T14:04:30.229744+0000","last_undegraded":"2024-04-25T14:04:30.229744+0000","last_fullsized":"2024-04-25T14:04:30.229744+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:17:31.862395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221934+0000","last_change":"2024-04-25T14:04:29.208196+0000","last_active":"2024-04-25T14:04:31.221934+0000","last_peered":"2024-04-25T14:04:31.221934+0000","last_clean":"2024-04-25T14:04:31.221934+0000","last_became_active":"2024-04-25T14:04:29.207877+0000","last_became_peered":"2024-04-25T14:04:29.207877+0000","last_unstale":"2024-04-25T14:04:31.221934+0000","last_undegraded":"2024-04-25T14:04:31.221934+0000","last_fullsized":"2024-04-25T14:04:31.221934+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:44.044045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225296+0000","last_change":"2024-04-25T14:04:30.218608+0000","last_active":"2024-04-25T14:04:30.225296+0000","last_peered":"2024-04-25T14:04:30.225296+0000","last_clean":"2024-04-25T14:04:30.225296+0000","last_became_active":"2024-04-25T14:04:30.217690+0000","last_became_peered":"2024-04-25T14:04:30.217690+0000","last_unstale":"2024-04-25T14:04:30.225296+0000","last_undegraded":"2024-04-25T14:04:30.225296+0000","last_fullsized":"2024-04-25T14:04:30.225296+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:17.744141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"55'8","reported_seq":20,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.232106+0000","last_change":"2024-04-25T14:04:29.219053+0000","last_active":"2024-04-25T14:04:30.232106+0000","last_peered":"2024-04-25T14:04:30.232106+0000","last_clean":"2024-04-25T14:04:30.232106+0000","last_became_active":"2024-04-25T14:04:29.217576+0000","last_became_peered":"2024-04-25T14:04:29.217576+0000","last_unstale":"2024-04-25T14:04:30.232106+0000","last_undegraded":"2024-04-25T14:04:30.232106+0000","last_fullsized":"2024-04-25T14:04:30.232106+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:25.792198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229369+0000","last_change":"2024-04-25T14:04:30.218916+0000","last_active":"2024-04-25T14:04:30.229369+0000","last_peered":"2024-04-25T14:04:30.229369+0000","last_clean":"2024-04-25T14:04:30.229369+0000","last_became_active":"2024-04-25T14:04:30.218466+0000","last_became_peered":"2024-04-25T14:04:30.218466+0000","last_unstale":"2024-04-25T14:04:30.229369+0000","last_undegraded":"2024-04-25T14:04:30.229369+0000","last_fullsized":"2024-04-25T14:04:30.229369+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"55'1","reported_seq":13,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.230228+0000","last_change":"2024-04-25T14:04:29.220594+0000","last_active":"2024-04-25T14:04:30.230228+0000","last_peered":"2024-04-25T14:04:30.230228+0000","last_clean":"2024-04-25T14:04:30.230228+0000","last_became_active":"2024-04-25T14:04:29.220302+0000","last_became_peered":"2024-04-25T14:04:29.220302+0000","last_unstale":"2024-04-25T14:04:30.230228+0000","last_undegraded":"2024-04-25T14:04:30.230228+0000","last_fullsized":"2024-04-25T14:04:30.230228+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T15:36:09.898546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231186+0000","last_change":"2024-04-25T14:04:30.227183+0000","last_active":"2024-04-25T14:04:30.231186+0000","last_peered":"2024-04-25T14:04:30.231186+0000","last_clean":"2024-04-25T14:04:30.231186+0000","last_became_active":"2024-04-25T14:04:30.226888+0000","last_became_peered":"2024-04-25T14:04:30.226888+0000","last_unstale":"2024-04-25T14:04:30.231186+0000","last_undegraded":"2024-04-25T14:04:30.231186+0000","last_fullsized":"2024-04-25T14:04:30.231186+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:13:45.046031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216930+0000","last_change":"2024-04-25T14:04:29.206123+0000","last_active":"2024-04-25T14:04:30.216930+0000","last_peered":"2024-04-25T14:04:30.216930+0000","last_clean":"2024-04-25T14:04:30.216930+0000","last_became_active":"2024-04-25T14:04:29.205802+0000","last_became_peered":"2024-04-25T14:04:29.205802+0000","last_unstale":"2024-04-25T14:04:30.216930+0000","last_undegraded":"2024-04-25T14:04:30.216930+0000","last_fullsized":"2024-04-25T14:04:30.216930+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:40:25.192007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220951+0000","last_change":"2024-04-25T14:04:30.226936+0000","last_active":"2024-04-25T14:04:31.220951+0000","last_peered":"2024-04-25T14:04:31.220951+0000","last_clean":"2024-04-25T14:04:31.220951+0000","last_became_active":"2024-04-25T14:04:30.226644+0000","last_became_peered":"2024-04-25T14:04:30.226644+0000","last_unstale":"2024-04-25T14:04:31.220951+0000","last_undegraded":"2024-04-25T14:04:31.220951+0000","last_fullsized":"2024-04-25T14:04:31.220951+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:18:45.625324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"55'1","reported_seq":15,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220791+0000","last_change":"2024-04-25T14:04:29.213637+0000","last_active":"2024-04-25T14:04:31.220791+0000","last_peered":"2024-04-25T14:04:31.220791+0000","last_clean":"2024-04-25T14:04:31.220791+0000","last_became_active":"2024-04-25T14:04:29.213288+0000","last_became_peered":"2024-04-25T14:04:29.213288+0000","last_unstale":"2024-04-25T14:04:31.220791+0000","last_undegraded":"2024-04-25T14:04:31.220791+0000","last_fullsized":"2024-04-25T14:04:31.220791+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:45:09.681772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222600+0000","last_change":"2024-04-25T14:04:30.220608+0000","last_active":"2024-04-25T14:04:30.222600+0000","last_peered":"2024-04-25T14:04:30.222600+0000","last_clean":"2024-04-25T14:04:30.222600+0000","last_became_active":"2024-04-25T14:04:30.220299+0000","last_became_peered":"2024-04-25T14:04:30.220299+0000","last_unstale":"2024-04-25T14:04:30.222600+0000","last_undegraded":"2024-04-25T14:04:30.222600+0000","last_fullsized":"2024-04-25T14:04:30.222600+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.221152+0000","last_change":"2024-04-25T14:04:29.215173+0000","last_active":"2024-04-25T14:04:31.221152+0000","last_peered":"2024-04-25T14:04:31.221152+0000","last_clean":"2024-04-25T14:04:31.221152+0000","last_became_active":"2024-04-25T14:04:29.214864+0000","last_became_peered":"2024-04-25T14:04:29.214864+0000","last_unstale":"2024-04-25T14:04:31.221152+0000","last_undegraded":"2024-04-25T14:04:31.221152+0000","last_fullsized":"2024-04-25T14:04:31.221152+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231257+0000","last_change":"2024-04-25T14:04:30.223667+0000","last_active":"2024-04-25T14:04:30.231257+0000","last_peered":"2024-04-25T14:04:30.231257+0000","last_clean":"2024-04-25T14:04:30.231257+0000","last_became_active":"2024-04-25T14:04:30.223339+0000","last_became_peered":"2024-04-25T14:04:30.223339+0000","last_unstale":"2024-04-25T14:04:30.231257+0000","last_undegraded":"2024-04-25T14:04:30.231257+0000","last_fullsized":"2024-04-25T14:04:30.231257+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:49:27.859138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"55'2","reported_seq":14,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.224655+0000","last_change":"2024-04-25T14:04:29.212470+0000","last_active":"2024-04-25T14:04:30.224655+0000","last_peered":"2024-04-25T14:04:30.224655+0000","last_clean":"2024-04-25T14:04:30.224655+0000","last_became_active":"2024-04-25T14:04:29.212178+0000","last_became_peered":"2024-04-25T14:04:29.212178+0000","last_unstale":"2024-04-25T14:04:30.224655+0000","last_undegraded":"2024-04-25T14:04:30.224655+0000","last_fullsized":"2024-04-25T14:04:30.224655+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:39.995488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220014+0000","last_change":"2024-04-25T14:04:30.223666+0000","last_active":"2024-04-25T14:04:31.220014+0000","last_peered":"2024-04-25T14:04:31.220014+0000","last_clean":"2024-04-25T14:04:31.220014+0000","last_became_active":"2024-04-25T14:04:30.223371+0000","last_became_peered":"2024-04-25T14:04:30.223371+0000","last_unstale":"2024-04-25T14:04:31.220014+0000","last_undegraded":"2024-04-25T14:04:31.220014+0000","last_fullsized":"2024-04-25T14:04:31.220014+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:42.631391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203144+0000","last_change":"2024-04-25T14:04:29.219331+0000","last_active":"2024-04-25T14:04:30.203144+0000","last_peered":"2024-04-25T14:04:30.203144+0000","last_clean":"2024-04-25T14:04:30.203144+0000","last_became_active":"2024-04-25T14:04:29.219039+0000","last_became_peered":"2024-04-25T14:04:29.219039+0000","last_unstale":"2024-04-25T14:04:30.203144+0000","last_undegraded":"2024-04-25T14:04:30.203144+0000","last_fullsized":"2024-04-25T14:04:30.203144+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:28.674456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231828+0000","last_change":"2024-04-25T14:04:30.230603+0000","last_active":"2024-04-25T14:04:30.231828+0000","last_peered":"2024-04-25T14:04:30.231828+0000","last_clean":"2024-04-25T14:04:30.231828+0000","last_became_active":"2024-04-25T14:04:30.230111+0000","last_became_peered":"2024-04-25T14:04:30.230111+0000","last_unstale":"2024-04-25T14:04:30.231828+0000","last_undegraded":"2024-04-25T14:04:30.231828+0000","last_fullsized":"2024-04-25T14:04:30.231828+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:50.366093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.222328+0000","last_change":"2024-04-25T14:04:29.217705+0000","last_active":"2024-04-25T14:04:30.222328+0000","last_peered":"2024-04-25T14:04:30.222328+0000","last_clean":"2024-04-25T14:04:30.222328+0000","last_became_active":"2024-04-25T14:04:29.217399+0000","last_became_peered":"2024-04-25T14:04:29.217399+0000","last_unstale":"2024-04-25T14:04:30.222328+0000","last_undegraded":"2024-04-25T14:04:30.222328+0000","last_fullsized":"2024-04-25T14:04:30.222328+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:28.674456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216029+0000","last_change":"2024-04-25T14:04:30.212096+0000","last_active":"2024-04-25T14:04:30.216029+0000","last_peered":"2024-04-25T14:04:30.216029+0000","last_clean":"2024-04-25T14:04:30.216029+0000","last_became_active":"2024-04-25T14:04:30.211801+0000","last_became_peered":"2024-04-25T14:04:30.211801+0000","last_unstale":"2024-04-25T14:04:30.216029+0000","last_undegraded":"2024-04-25T14:04:30.216029+0000","last_fullsized":"2024-04-25T14:04:30.216029+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203330+0000","last_change":"2024-04-25T14:04:29.209296+0000","last_active":"2024-04-25T14:04:30.203330+0000","last_peered":"2024-04-25T14:04:30.203330+0000","last_clean":"2024-04-25T14:04:30.203330+0000","last_became_active":"2024-04-25T14:04:29.208978+0000","last_became_peered":"2024-04-25T14:04:29.208978+0000","last_unstale":"2024-04-25T14:04:30.203330+0000","last_undegraded":"2024-04-25T14:04:30.203330+0000","last_fullsized":"2024-04-25T14:04:30.203330+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:07:34.634391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.217192+0000","last_change":"2024-04-25T14:04:30.216153+0000","last_active":"2024-04-25T14:04:30.217192+0000","last_peered":"2024-04-25T14:04:30.217192+0000","last_clean":"2024-04-25T14:04:30.217192+0000","last_became_active":"2024-04-25T14:04:30.211851+0000","last_became_peered":"2024-04-25T14:04:30.211851+0000","last_unstale":"2024-04-25T14:04:30.217192+0000","last_undegraded":"2024-04-25T14:04:30.217192+0000","last_fullsized":"2024-04-25T14:04:30.217192+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:26.794183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"55'2","reported_seq":16,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220898+0000","last_change":"2024-04-25T14:04:29.223011+0000","last_active":"2024-04-25T14:04:31.220898+0000","last_peered":"2024-04-25T14:04:31.220898+0000","last_clean":"2024-04-25T14:04:31.220898+0000","last_became_active":"2024-04-25T14:04:29.222677+0000","last_became_peered":"2024-04-25T14:04:29.222677+0000","last_unstale":"2024-04-25T14:04:31.220898+0000","last_undegraded":"2024-04-25T14:04:31.220898+0000","last_fullsized":"2024-04-25T14:04:31.220898+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:35:51.800914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.215967+0000","last_change":"2024-04-25T14:04:30.206536+0000","last_active":"2024-04-25T14:04:30.215967+0000","last_peered":"2024-04-25T14:04:30.215967+0000","last_clean":"2024-04-25T14:04:30.215967+0000","last_became_active":"2024-04-25T14:04:30.206254+0000","last_became_peered":"2024-04-25T14:04:30.206254+0000","last_unstale":"2024-04-25T14:04:30.215967+0000","last_undegraded":"2024-04-25T14:04:30.215967+0000","last_fullsized":"2024-04-25T14:04:30.215967+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:53:25.133215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220710+0000","last_change":"2024-04-25T14:04:29.208767+0000","last_active":"2024-04-25T14:04:31.220710+0000","last_peered":"2024-04-25T14:04:31.220710+0000","last_clean":"2024-04-25T14:04:31.220710+0000","last_became_active":"2024-04-25T14:04:29.207051+0000","last_became_peered":"2024-04-25T14:04:29.207051+0000","last_unstale":"2024-04-25T14:04:31.220710+0000","last_undegraded":"2024-04-25T14:04:31.220710+0000","last_fullsized":"2024-04-25T14:04:31.220710+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:34:15.406391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229312+0000","last_change":"2024-04-25T14:04:30.218526+0000","last_active":"2024-04-25T14:04:30.229312+0000","last_peered":"2024-04-25T14:04:30.229312+0000","last_clean":"2024-04-25T14:04:30.229312+0000","last_became_active":"2024-04-25T14:04:30.218230+0000","last_became_peered":"2024-04-25T14:04:30.218230+0000","last_unstale":"2024-04-25T14:04:30.229312+0000","last_undegraded":"2024-04-25T14:04:30.229312+0000","last_fullsized":"2024-04-25T14:04:30.229312+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:39:26.751098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.207563+0000","last_change":"2024-04-25T14:04:29.217799+0000","last_active":"2024-04-25T14:04:30.207563+0000","last_peered":"2024-04-25T14:04:30.207563+0000","last_clean":"2024-04-25T14:04:30.207563+0000","last_became_active":"2024-04-25T14:04:29.217428+0000","last_became_peered":"2024-04-25T14:04:29.217428+0000","last_unstale":"2024-04-25T14:04:30.207563+0000","last_undegraded":"2024-04-25T14:04:30.207563+0000","last_fullsized":"2024-04-25T14:04:30.207563+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:54:41.629406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.225586+0000","last_change":"2024-04-25T14:04:30.220410+0000","last_active":"2024-04-25T14:04:30.225586+0000","last_peered":"2024-04-25T14:04:30.225586+0000","last_clean":"2024-04-25T14:04:30.225586+0000","last_became_active":"2024-04-25T14:04:30.220120+0000","last_became_peered":"2024-04-25T14:04:30.220120+0000","last_unstale":"2024-04-25T14:04:30.225586+0000","last_undegraded":"2024-04-25T14:04:30.225586+0000","last_fullsized":"2024-04-25T14:04:30.225586+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:06:33.189618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.203311+0000","last_change":"2024-04-25T14:04:29.219179+0000","last_active":"2024-04-25T14:04:30.203311+0000","last_peered":"2024-04-25T14:04:30.203311+0000","last_clean":"2024-04-25T14:04:30.203311+0000","last_became_active":"2024-04-25T14:04:29.218839+0000","last_became_peered":"2024-04-25T14:04:29.218839+0000","last_unstale":"2024-04-25T14:04:30.203311+0000","last_undegraded":"2024-04-25T14:04:30.203311+0000","last_fullsized":"2024-04-25T14:04:30.203311+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:00:12.979944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.226284+0000","last_change":"2024-04-25T14:04:30.221377+0000","last_active":"2024-04-25T14:04:30.226284+0000","last_peered":"2024-04-25T14:04:30.226284+0000","last_clean":"2024-04-25T14:04:30.226284+0000","last_became_active":"2024-04-25T14:04:30.216730+0000","last_became_peered":"2024-04-25T14:04:30.216730+0000","last_unstale":"2024-04-25T14:04:30.226284+0000","last_undegraded":"2024-04-25T14:04:30.226284+0000","last_fullsized":"2024-04-25T14:04:30.226284+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T14:08:29.676442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.202738+0000","last_change":"2024-04-25T14:04:29.214185+0000","last_active":"2024-04-25T14:04:30.202738+0000","last_peered":"2024-04-25T14:04:30.202738+0000","last_clean":"2024-04-25T14:04:30.202738+0000","last_became_active":"2024-04-25T14:04:29.213861+0000","last_became_peered":"2024-04-25T14:04:29.213861+0000","last_unstale":"2024-04-25T14:04:30.202738+0000","last_undegraded":"2024-04-25T14:04:30.202738+0000","last_fullsized":"2024-04-25T14:04:30.202738+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:20:16.742155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.231378+0000","last_change":"2024-04-25T14:04:30.226343+0000","last_active":"2024-04-25T14:04:30.231378+0000","last_peered":"2024-04-25T14:04:30.231378+0000","last_clean":"2024-04-25T14:04:30.231378+0000","last_became_active":"2024-04-25T14:04:30.225993+0000","last_became_peered":"2024-04-25T14:04:30.225993+0000","last_unstale":"2024-04-25T14:04:30.231378+0000","last_undegraded":"2024-04-25T14:04:30.231378+0000","last_fullsized":"2024-04-25T14:04:30.231378+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:54:23.991464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220864+0000","last_change":"2024-04-25T14:04:29.210709+0000","last_active":"2024-04-25T14:04:31.220864+0000","last_peered":"2024-04-25T14:04:31.220864+0000","last_clean":"2024-04-25T14:04:31.220864+0000","last_became_active":"2024-04-25T14:04:29.210390+0000","last_became_peered":"2024-04-25T14:04:29.210390+0000","last_unstale":"2024-04-25T14:04:31.220864+0000","last_undegraded":"2024-04-25T14:04:31.220864+0000","last_fullsized":"2024-04-25T14:04:31.220864+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:07:11.326588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.229262+0000","last_change":"2024-04-25T14:04:30.211440+0000","last_active":"2024-04-25T14:04:30.229262+0000","last_peered":"2024-04-25T14:04:30.229262+0000","last_clean":"2024-04-25T14:04:30.229262+0000","last_became_active":"2024-04-25T14:04:30.211139+0000","last_became_peered":"2024-04-25T14:04:30.211139+0000","last_unstale":"2024-04-25T14:04:30.229262+0000","last_undegraded":"2024-04-25T14:04:30.229262+0000","last_fullsized":"2024-04-25T14:04:30.229262+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:48.721725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.206648+0000","last_change":"2024-04-25T14:04:29.208231+0000","last_active":"2024-04-25T14:04:30.206648+0000","last_peered":"2024-04-25T14:04:30.206648+0000","last_clean":"2024-04-25T14:04:30.206648+0000","last_became_active":"2024-04-25T14:04:29.207908+0000","last_became_peered":"2024-04-25T14:04:29.207908+0000","last_unstale":"2024-04-25T14:04:30.206648+0000","last_undegraded":"2024-04-25T14:04:30.206648+0000","last_fullsized":"2024-04-25T14:04:30.206648+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:21:36.915963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.219919+0000","last_change":"2024-04-25T14:04:30.221325+0000","last_active":"2024-04-25T14:04:31.219919+0000","last_peered":"2024-04-25T14:04:31.219919+0000","last_clean":"2024-04-25T14:04:31.219919+0000","last_became_active":"2024-04-25T14:04:30.220805+0000","last_became_peered":"2024-04-25T14:04:30.220805+0000","last_unstale":"2024-04-25T14:04:31.219919+0000","last_undegraded":"2024-04-25T14:04:31.219919+0000","last_fullsized":"2024-04-25T14:04:31.219919+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:29.187925+0000","last_clean_scrub_stamp":"2024-04-25T14:04:29.187925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:38.597579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":55,"state":"active+clean","last_fresh":"2024-04-25T14:04:30.216996+0000","last_change":"2024-04-25T14:04:29.219631+0000","last_active":"2024-04-25T14:04:30.216996+0000","last_peered":"2024-04-25T14:04:30.216996+0000","last_clean":"2024-04-25T14:04:30.216996+0000","last_became_active":"2024-04-25T14:04:29.218130+0000","last_became_peered":"2024-04-25T14:04:29.218130+0000","last_unstale":"2024-04-25T14:04:30.216996+0000","last_undegraded":"2024-04-25T14:04:30.216996+0000","last_fullsized":"2024-04-25T14:04:30.216996+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:04:28.185939+0000","last_clean_scrub_stamp":"2024-04-25T14:04:28.185939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:08.521825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":183,"reported_epoch":56,"state":"active+clean","last_fresh":"2024-04-25T14:04:31.220049+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:04:31.220049+0000","last_peered":"2024-04-25T14:04:31.220049+0000","last_clean":"2024-04-25T14:04:31.220049+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:04:31.220049+0000","last_undegraded":"2024-04-25T14:04:31.220049+0000","last_fullsized":"2024-04-25T14:04:31.220049+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28264,"kb_used_data":1240,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343576,"statfs":{"total":96636764160,"available":96607821824,"internally_reserved":0,"allocated":1269760,"data_stored":894013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344160,"statfs":{"total":96636764160,"available":96608419840,"internally_reserved":0,"allocated":675840,"data_stored":304173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":655360,"data_stored":296829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":659456,"data_stored":297411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":671744,"data_stored":298463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574883,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":667648,"data_stored":298441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":663552,"data_stored":296989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28248,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343592,"statfs":{"total":96636764160,"available":96607838208,"internally_reserved":0,"allocated":1257472,"data_stored":887253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:04:33.569 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T14:04:33.569 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-25T14:04:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.720+0000 7fa2daec6640 1 Processor -- start 2024-04-25T14:04:33.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.720+0000 7fa2daec6640 1 -- start start 2024-04-25T14:04:33.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2daec6640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2d4070820 0x7fa2d41609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2daec6640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2d40cf300 con 0x7fa2d40d03a0 2024-04-25T14:04:33.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2daec6640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2d40cf5e0 con 0x7fa2d40cdca0 2024-04-25T14:04:33.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2daec6640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa2d40cff10 con 0x7fa2d4070820 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2d4070820 0x7fa2d41609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d8c62640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa2d40d03a0 0x7fa2d40cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41206/0 (socket says 172.21.15.150:41206) 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d8c62640 1 -- 172.21.15.150:0/2030524202 learned_addr learned my addr 172.21.15.150:0/2030524202 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d3fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2d4070820 0x7fa2d41609e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59980/0 (socket says 172.21.15.150:59980) 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d3fff640 1 -- 172.21.15.150:0/2030524202 >> v1:172.21.15.187:6789/0 conn(0x7fa2d40cdca0 legacy=0x7fa2d406f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d3fff640 1 -- 172.21.15.150:0/2030524202 >> v1:172.21.15.150:6789/0 conn(0x7fa2d40d03a0 legacy=0x7fa2d40cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.725 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d37fe640 1 -- 172.21.15.150:0/2030524202 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2032684051 0 0) 0x7fa2d40cf300 con 0x7fa2d40d03a0 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.722+0000 7fa2d3fff640 1 -- 172.21.15.150:0/2030524202 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d40cf8c0 con 0x7fa2d4070820 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.723+0000 7fa2d3fff640 1 --2- 172.21.15.150:0/2030524202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d4070820 0x7fa2d41609e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80097a0 tx=0x7fa2c802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daedfd5511712aa0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.723+0000 7fa2d37fe640 1 -- 172.21.15.150:0/2030524202 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8037070 con 0x7fa2d4070820 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.723+0000 7fa2d37fe640 1 -- 172.21.15.150:0/2030524202 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2c8032440 con 0x7fa2d4070820 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.723+0000 7fa2d37fe640 1 -- 172.21.15.150:0/2030524202 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8032810 con 0x7fa2d4070820 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 -- 172.21.15.150:0/2030524202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d4070820 msgr2=0x7fa2d41609e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2030524202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d4070820 0x7fa2d41609e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fa2c80097a0 tx=0x7fa2c802a230 comp rx=0 tx=0).stop 2024-04-25T14:04:33.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 -- 172.21.15.150:0/2030524202 shutdown_connections 2024-04-25T14:04:33.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2030524202 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d4070820 0x7fa2d41609e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 -- 172.21.15.150:0/2030524202 >> 172.21.15.150:0/2030524202 conn(0x7fa2d40ce390 msgr2=0x7fa2d40ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 -- 172.21.15.150:0/2030524202 shutdown_connections 2024-04-25T14:04:33.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.724+0000 7fa2daec6640 1 -- 172.21.15.150:0/2030524202 wait complete. 2024-04-25T14:04:33.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.725+0000 7fa2daec6640 1 Processor -- start 2024-04-25T14:04:33.731 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.726+0000 7fa2daec6640 1 -- start start 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2d4070820 0x7fa2d40b0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2d40cdca0 0x7fa2d4150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d41610a0 con 0x7fa2d40d03a0 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2d4160f20 con 0x7fa2d4070820 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2daec6640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2d4161220 con 0x7fa2d40cdca0 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:59990/0 (socket says 172.21.15.150:59990) 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 -- 172.21.15.150:0/2788611078 learned_addr learned my addr 172.21.15.150:0/2788611078 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d8c62640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2d4070820 0x7fa2d40b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d3fff640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2d40cdca0 0x7fa2d4150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 -- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2d40cdca0 msgr2=0x7fa2d4150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2d40cdca0 0x7fa2d4150d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.732 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 -- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2d4070820 msgr2=0x7fa2d40b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2d4070820 0x7fa2d40b0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.727+0000 7fa2d9463640 1 -- 172.21.15.150:0/2788611078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2d40af200 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2d9463640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c6b0 tx=0x7fa2cc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc003b70 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2cc003d10 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2d41e6a70 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2cc005310 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.728+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2d41e7030 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.729+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2d4076a40 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.729+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa2cc0054b0 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.730+0000 7fa2d17fa640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 0x7fa2c00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.730+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fa2cc095050 con 0x7fa2d40d03a0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.730+0000 7fa2d8c62640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 0x7fa2c00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:33.733 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.730+0000 7fa2d8c62640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 0x7fa2c00769f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4000f60 tx=0x7fa2c4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:33.734 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.734+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2cc05f770 con 0x7fa2d40d03a0 2024-04-25T14:04:33.974 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.975+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fa2d40b3030 con 0x7fa2d40d03a0 2024-04-25T14:04:33.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.977+0000 7fa2d17fa640 1 -- 172.21.15.150:0/2788611078 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) ==== 190+0+147 (secure 0 0 0) 0x7fa2cc05f070 con 0x7fa2d40d03a0 2024-04-25T14:04:33.976 INFO:teuthology.orchestra.run.smithi150.stdout:[client.0] 2024-04-25T14:04:33.976 INFO:teuthology.orchestra.run.smithi150.stdout: key = AQAnYipmZQMOBBAA+IINh20YgNeF+b/8BY75jQ== 2024-04-25T14:04:33.976 INFO:teuthology.orchestra.run.smithi150.stdout: caps mds = "allow" 2024-04-25T14:04:33.977 INFO:teuthology.orchestra.run.smithi150.stdout: caps mon = "allow r" 2024-04-25T14:04:33.977 INFO:teuthology.orchestra.run.smithi150.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-25T14:04:33.977 INFO:teuthology.orchestra.run.smithi150.stderr:updated caps for client.0 2024-04-25T14:04:33.978 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 msgr2=0x7fa2c00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 0x7fa2c00769f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa2c4000f60 tx=0x7fa2c4009040 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 msgr2=0x7fa2d41579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc00c6b0 tx=0x7fa2cc00cb70 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 shutdown_connections 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2c00744e0 0x7fa2c00769f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2d40d03a0 0x7fa2d41579f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2d40cdca0 0x7fa2d4150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 --2- 172.21.15.150:0/2788611078 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2d4070820 0x7fa2d40b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 >> 172.21.15.150:0/2788611078 conn(0x7fa2d40ce390 msgr2=0x7fa2d415d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 shutdown_connections 2024-04-25T14:04:33.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:33.979+0000 7fa2daec6640 1 -- 172.21.15.150:0/2788611078 wait complete. 2024-04-25T14:04:33.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:34.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.098+0000 7fa5df59e640 1 Processor -- start 2024-04-25T14:04:34.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.098+0000 7fa5df59e640 1 -- start start 2024-04-25T14:04:34.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5df59e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa5e00d4520 0x7fa5e0160690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5df59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5e0160bd0 0x7fa5e006cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5df59e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5e00d4d00 con 0x7fa5e00d3d30 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5df59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa5e00d4140 con 0x7fa5e0160bd0 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5df59e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa5e006c7a0 con 0x7fa5e00d4520 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5e0160bd0 0x7fa5e006cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa5e0160bd0 0x7fa5e006cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60002/0 (socket says 172.21.15.150:60002) 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 -- 172.21.15.150:0/2210887014 learned_addr learned my addr 172.21.15.150:0/2210887014 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ddd9b640 1 --2- 172.21.15.150:0/2210887014 >> v2:172.21.15.187:3300/0 conn(0x7fa5e00d4520 0x7fa5e0160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 -- 172.21.15.150:0/2210887014 >> v2:172.21.15.187:3300/0 conn(0x7fa5e00d4520 msgr2=0x7fa5e0160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 --2- 172.21.15.150:0/2210887014 >> v2:172.21.15.187:3300/0 conn(0x7fa5e00d4520 0x7fa5e0160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 -- 172.21.15.150:0/2210887014 >> v1:172.21.15.187:6789/0 conn(0x7fa5e00d3d30 legacy=0x7fa5e00b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.100+0000 7fa5ded9d640 1 -- 172.21.15.150:0/2210887014 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e00d4fe0 con 0x7fa5e0160bd0 2024-04-25T14:04:34.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.101+0000 7fa5ded9d640 1 --2- 172.21.15.150:0/2210887014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006cc30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa5e008ee50 tx=0x7fa5d802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=644b204e6f413150 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.101+0000 7fa5dd59a640 1 -- 172.21.15.150:0/2210887014 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8037070 con 0x7fa5e0160bd0 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.101+0000 7fa5dd59a640 1 -- 172.21.15.150:0/2210887014 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d8032440 con 0x7fa5e0160bd0 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.101+0000 7fa5dd59a640 1 -- 172.21.15.150:0/2210887014 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8032810 con 0x7fa5e0160bd0 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 -- 172.21.15.150:0/2210887014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e0160bd0 msgr2=0x7fa5e006cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2210887014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006cc30 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa5e008ee50 tx=0x7fa5d802a270 comp rx=0 tx=0).stop 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 -- 172.21.15.150:0/2210887014 shutdown_connections 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2210887014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006cc30 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2210887014 >> v2:172.21.15.187:3300/0 conn(0x7fa5e00d4520 0x7fa5e0160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 -- 172.21.15.150:0/2210887014 >> 172.21.15.150:0/2210887014 conn(0x7fa5e00b7440 msgr2=0x7fa5e00b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 -- 172.21.15.150:0/2210887014 shutdown_connections 2024-04-25T14:04:34.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.102+0000 7fa5df59e640 1 -- 172.21.15.150:0/2210887014 wait complete. 2024-04-25T14:04:34.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.104+0000 7fa5df59e640 1 Processor -- start 2024-04-25T14:04:34.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.104+0000 7fa5df59e640 1 -- start start 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e00d4520 0x7fa5e006e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e00d4930 con 0x7fa5e00d3d30 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e0160fe0 con 0x7fa5e0160bd0 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5df59e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e00d4ab0 con 0x7fa5e00d4520 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5de59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5de59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60004/0 (socket says 172.21.15.150:60004) 2024-04-25T14:04:34.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5de59c640 1 -- 172.21.15.150:0/2611780587 learned_addr learned my addr 172.21.15.150:0/2611780587 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.105+0000 7fa5ded9d640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5ddd9b640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e00d4520 0x7fa5e006e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 -- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e00d4520 msgr2=0x7fa5e006e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e00d4520 0x7fa5e006e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 -- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 msgr2=0x7fa5e006ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 -- 172.21.15.150:0/2611780587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e0071af0 con 0x7fa5e00d3d30 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5ded9d640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5de59c640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000c730 tx=0x7fa5d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0016020 con 0x7fa5e00d3d30 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5d0006d50 con 0x7fa5e00d3d30 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5d0003bb0 con 0x7fa5e00d3d30 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5e0071dd0 con 0x7fa5e00d3d30 2024-04-25T14:04:34.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.106+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5e0072390 con 0x7fa5e00d3d30 2024-04-25T14:04:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.107+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5e0050740 con 0x7fa5e00d3d30 2024-04-25T14:04:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.108+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa5d000a850 con 0x7fa5e00d3d30 2024-04-25T14:04:34.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.108+0000 7fa5ceffd640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 0x7fa5c00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.108+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fa5d0095000 con 0x7fa5e00d3d30 2024-04-25T14:04:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.108+0000 7fa5ddd9b640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 0x7fa5c00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.109 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.109+0000 7fa5ddd9b640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 0x7fa5c00769f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4006fd0 tx=0x7fa5d4008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.113+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5d0020020 con 0x7fa5e00d3d30 2024-04-25T14:04:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.323+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5e01603b0 con 0x7fa5e00d3d30 2024-04-25T14:04:34.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.324+0000 7fa5ceffd640 1 -- 172.21.15.150:0/2611780587 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5187 (secure 0 0 0) 0x7fa5d005f690 con 0x7fa5e00d3d30 2024-04-25T14:04:34.324 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:34.324 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/468558883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":468558883},{"type":"v1","addr":"172.21.15.187:6834","nonce":468558883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3011235888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3011235888},{"type":"v1","addr":"172.21.15.150:6835","nonce":3011235888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5823,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1873279280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1873279280},{"type":"v1","addr":"172.21.15.150:6837","nonce":1873279280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:31.564985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5832},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5832":{"gid":5832,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/133937444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":133937444},{"type":"v1","addr":"172.21.15.187:6835","nonce":133937444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5832,"qdb_cluster":[5832]},"id":3}]} 2024-04-25T14:04:34.324 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 30 2024-04-25T14:04:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 msgr2=0x7fa5c00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 0x7fa5c00769f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4006fd0 tx=0x7fa5d4008040 comp rx=0 tx=0).stop 2024-04-25T14:04:34.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 msgr2=0x7fa5e00711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa5d000c730 tx=0x7fa5d000cbf0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 shutdown_connections 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa5c00744e0 0x7fa5c00769f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa5e0160bd0 0x7fa5e006ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa5e00d4520 0x7fa5e006e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 --2- 172.21.15.150:0/2611780587 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa5e00d3d30 0x7fa5e00711f0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 >> 172.21.15.150:0/2611780587 conn(0x7fa5e00b7440 msgr2=0x7fa5e015fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 shutdown_connections 2024-04-25T14:04:34.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.326+0000 7fa5df59e640 1 -- 172.21.15.150:0/2611780587 wait complete. 2024-04-25T14:04:34.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:04:29.564574+0000', 'modified': '2024-04-25T14:04:31.564985+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5832}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5832': {'gid': 5832, 'name': 'd', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/133937444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 133937444}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 133937444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5832, 'qdb_cluster': [5832]} 2024-04-25T14:04:34.739 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:04:34.739 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T14:04:34.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.850+0000 7f5690fce640 1 Processor -- start 2024-04-25T14:04:34.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.851+0000 7f5690fce640 1 -- start start 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f5690fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f5690fce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f568c06d800 0x7f568c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f5690fce640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f568c0b9600 con 0x7f568c0b91f0 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f5690fce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f568c06cc60 con 0x7f568c06d800 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f5690fce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f568c0d4f20 con 0x7f568c0b4c10 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f568ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f568c06d800 0x7f568c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f568ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f568c06d800 0x7f568c0d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60018/0 (socket says 172.21.15.150:60018) 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f568ad9d640 1 -- 172.21.15.150:0/3355395639 learned_addr learned my addr 172.21.15.150:0/3355395639 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.852+0000 7f568a59c640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568ad9d640 1 -- 172.21.15.150:0/3355395639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 msgr2=0x7f568c160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568ad9d640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568ad9d640 1 -- 172.21.15.150:0/3355395639 >> v1:172.21.15.187:6789/0 conn(0x7f568c0b91f0 legacy=0x7f568c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568ad9d640 1 -- 172.21.15.150:0/3355395639 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c0d5200 con 0x7f568c06d800 2024-04-25T14:04:34.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568a59c640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568ad9d640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c06d800 0x7f568c0d0950 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f568c06cde0 tx=0x7f568402a200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c323d6fdbf87e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568959a640 1 -- 172.21.15.150:0/3355395639 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684037070 con 0x7f568c06d800 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.853+0000 7f568959a640 1 -- 172.21.15.150:0/3355395639 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684032440 con 0x7f568c06d800 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f568959a640 1 -- 172.21.15.150:0/3355395639 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684032810 con 0x7f568c06d800 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 -- 172.21.15.150:0/3355395639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c06d800 msgr2=0x7f568c0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c06d800 0x7f568c0d0950 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f568c06cde0 tx=0x7f568402a200 comp rx=0 tx=0).stop 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 -- 172.21.15.150:0/3355395639 shutdown_connections 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c06d800 0x7f568c0d0950 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 --2- 172.21.15.150:0/3355395639 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 -- 172.21.15.150:0/3355395639 >> 172.21.15.150:0/3355395639 conn(0x7f568c0b7690 msgr2=0x7f568c0b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 -- 172.21.15.150:0/3355395639 shutdown_connections 2024-04-25T14:04:34.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.854+0000 7f5690fce640 1 -- 172.21.15.150:0/3355395639 wait complete. 2024-04-25T14:04:34.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.856+0000 7f5690fce640 1 Processor -- start 2024-04-25T14:04:34.855 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.856+0000 7f5690fce640 1 -- start start 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.857+0000 7f5690fce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f568c06d800 0x7f568c0cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.857+0000 7f5690fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c0ac850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f568c06c8f0 con 0x7f568c0b91f0 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f568c06c610 con 0x7f568c06d800 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f568c06dc10 con 0x7f568c0b4c10 2024-04-25T14:04:34.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60030/0 (socket says 172.21.15.150:60030) 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 -- 172.21.15.150:0/2978975904 learned_addr learned my addr 172.21.15.150:0/2978975904 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568a59c640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f568c06d800 0x7f568c0cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c0ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 -- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 msgr2=0x7f568c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c0ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 -- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f568c06d800 msgr2=0x7f568c0cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f568c06d800 0x7f568c0cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 -- 172.21.15.150:0/2978975904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568c0ade70 con 0x7f568c0b91f0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f568ad9d640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f568402a240 tx=0x7f56840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f567affd640 1 -- 172.21.15.150:0/2978975904 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684037040 con 0x7f568c0b91f0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f567affd640 1 -- 172.21.15.150:0/2978975904 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684042070 con 0x7f568c0b91f0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568c0aef20 con 0x7f568c0b91f0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f567affd640 1 -- 172.21.15.150:0/2978975904 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5684030c90 con 0x7f568c0b91f0 2024-04-25T14:04:34.859 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.858+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568c0ad8b0 con 0x7f568c0b91f0 2024-04-25T14:04:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.860+0000 7f567affd640 1 -- 172.21.15.150:0/2978975904 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f568402ad10 con 0x7f568c0b91f0 2024-04-25T14:04:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.861+0000 7f567affd640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 0x7f566c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.861+0000 7f568a59c640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 0x7f566c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.861+0000 7f567affd640 1 -- 172.21.15.150:0/2978975904 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f56840b5b40 con 0x7f568c0b91f0 2024-04-25T14:04:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.861+0000 7f568a59c640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 0x7f566c0769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f567c00f500 tx=0x7f567c007770 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.862+0000 7f5697e7d740 1 Processor -- start 2024-04-25T14:04:34.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.862+0000 7f5697e7d740 1 -- start start 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127552600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a127528260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a12748c740 con 0x55a12715d3b0 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a1274283a0 con 0x55a127186270 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a127488700 con 0x55a127165590 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60042/0 (socket says 172.21.15.150:60042) 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 -- 172.21.15.150:0/2654645137 learned_addr learned my addr 172.21.15.150:0/2654645137 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f568ad9d640 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a127528260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f568a59c640 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127552600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 -- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 msgr2=0x55a127528260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a127528260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 -- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 msgr2=0x55a127552600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127552600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.863+0000 7f5689d9b640 1 -- 172.21.15.150:0/2654645137 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a127444750 con 0x55a12715d3b0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.864+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5680009960 tx=0x7f5680009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1bca04ce69080c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.864+0000 7f566bfff640 1 -- 172.21.15.150:0/2654645137 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680017070 con 0x55a12715d3b0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.864+0000 7f566bfff640 1 -- 172.21.15.150:0/2654645137 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568000ea20 con 0x55a12715d3b0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.864+0000 7f566bfff640 1 -- 172.21.15.150:0/2654645137 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568000edd0 con 0x55a12715d3b0 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 -- 172.21.15.150:0/2654645137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 msgr2=0x55a12754efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f5680009960 tx=0x7f5680009e20 comp rx=0 tx=0).stop 2024-04-25T14:04:34.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 -- 172.21.15.150:0/2654645137 shutdown_connections 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a127528260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12754efa0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2654645137 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127552600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 -- 172.21.15.150:0/2654645137 >> 172.21.15.150:0/2654645137 conn(0x55a12717a3c0 msgr2=0x55a1275279f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 -- 172.21.15.150:0/2654645137 shutdown_connections 2024-04-25T14:04:34.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.865+0000 7f5697e7d740 1 -- 172.21.15.150:0/2654645137 wait complete. 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.866+0000 7f5697e7d740 1 -- start start 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.866+0000 7f5697e7d740 1 Processor -- start 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a12715d3b0 0x55a1275511a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127550b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a127413fe0 con 0x55a127165590 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a127555b40 con 0x55a127186270 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a1275452a0 con 0x55a12715d3b0 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5697e7d740 10 client.0 _ll_register_callbacks cb 0x55a12742a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5689d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5689d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60048/0 (socket says 172.21.15.150:60048) 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5689d9b640 1 -- 172.21.15.150:0/471949970 learned_addr learned my addr 172.21.15.150:0/471949970 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f568ad9d640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127550b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f568a59c640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a12715d3b0 0x55a1275511a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5689d9b640 1 -- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a12715d3b0 msgr2=0x55a1275511a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.867+0000 7f5689d9b640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a12715d3b0 0x55a1275511a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5689d9b640 1 -- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 msgr2=0x55a127550b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5689d9b640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127550b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5689d9b640 1 -- 172.21.15.150:0/471949970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a12719e2d0 con 0x55a127165590 2024-04-25T14:04:34.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f568ad9d640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127550b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5689d9b640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5680016000 tx=0x7f5680004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1e7ae653b39c1a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5680017040 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f5680027c30 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f568000ea20 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5680027e30 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 10 client.5955 fetch_fsmap learned FSMap version 30 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x55a1272686b0 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f568000ea20 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f56800037f0 con 0x55a127165590 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 10 client.5955 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 10 client.5955 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 20 client.5955 populate_metadata read hostname 'smithi150' 2024-04-25T14:04:34.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a127144e40 0x55a127554b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 4 client.5955 mds_command: new command op to 5832 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.868+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 --> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a12749a350 con 0x55a127144e40 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.869+0000 7f568a59c640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a127144e40 0x55a127554b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.869+0000 7f56697fa640 10 client.5955 ms_handle_connect on v2:172.21.15.187:6833/133937444 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.869+0000 7f568a59c640 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a127144e40 0x55a127554b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bd1b15bd861c3f8 server_cookie=37a776fd484cda2a in_seq=0 out_seq=0 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f56697fa640 1 -- 172.21.15.150:0/471949970 <== mds.0 v2:172.21.15.187:6833/133937444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a12749a350 con 0x55a127144e40 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f56697fa640 10 client.5955 handle_command_reply: tid=0 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 client.5955 shutdown 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 msgr2=0x55a1275505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5680016000 tx=0x7f5680004680 comp rx=0 tx=0).stop 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 shutdown_connections 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a127144e40 0x55a127554b00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127186270 0x55a127550b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127165590 0x55a1275505c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 --2- 172.21.15.150:0/471949970 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a12715d3b0 0x55a1275511a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f56697fa640 0 client.5955 ms_handle_reset on v2:172.21.15.187:6833/133937444 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 >> 172.21.15.150:0/471949970 conn(0x55a12717a3c0 msgr2=0x55a1275279f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 shutdown_connections 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 1 -- 172.21.15.150:0/471949970 wait complete. 2024-04-25T14:04:34.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.870+0000 7f5697e7d740 20 client.5955 trim_cache size 0 max 16384 2024-04-25T14:04:34.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.889+0000 7f5697e7d740 1 Processor -- start 2024-04-25T14:04:34.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.889+0000 7f5697e7d740 1 -- start start 2024-04-25T14:04:34.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a12715d3b0 0x55a127551a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a1275527f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f568c2022e0 con 0x55a127186270 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f568c0d0da0 con 0x55a12715d3b0 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.890+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f568c0d13f0 con 0x55a127165590 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60056/0 (socket says 172.21.15.150:60056) 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 -- 172.21.15.150:0/1209702789 learned_addr learned my addr 172.21.15.150:0/1209702789 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 -- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 msgr2=0x55a1275527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a1275527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 -- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a12715d3b0 msgr2=0x55a127551a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a12715d3b0 0x55a127551a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 -- 172.21.15.150:0/1209702789 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a127522440 con 0x55a127186270 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f568ad9d640 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5684006f40 tx=0x7f568407f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a4deed37be7aea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f566b7fe640 1 -- 172.21.15.150:0/1209702789 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568407f720 con 0x55a127186270 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f566b7fe640 1 -- 172.21.15.150:0/1209702789 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684099070 con 0x55a127186270 2024-04-25T14:04:34.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.891+0000 7f566b7fe640 1 -- 172.21.15.150:0/1209702789 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568407fc10 con 0x55a127186270 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 -- 172.21.15.150:0/1209702789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 msgr2=0x55a1275e3da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f5684006f40 tx=0x7f568407f620 comp rx=0 tx=0).stop 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 -- 172.21.15.150:0/1209702789 shutdown_connections 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a127186270 0x55a1275e3da0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127165590 0x55a1275527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 --2- 172.21.15.150:0/1209702789 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a12715d3b0 0x55a127551a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 -- 172.21.15.150:0/1209702789 >> 172.21.15.150:0/1209702789 conn(0x55a12717a3c0 msgr2=0x55a1275510f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 -- 172.21.15.150:0/1209702789 shutdown_connections 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.892+0000 7f5697e7d740 1 -- 172.21.15.150:0/1209702789 wait complete. 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.893+0000 7f5697e7d740 1 -- start start 2024-04-25T14:04:34.893 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.893+0000 7f5697e7d740 1 Processor -- start 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 0x55a127554ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127186270 0x55a127555010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55a1275532a0 con 0x55a12715d3b0 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55a12756ca60 con 0x55a127165590 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55a12756b690 con 0x55a127186270 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5697e7d740 10 client.0 _ll_register_callbacks cb 0x7f5680009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568a59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60060/0 (socket says 172.21.15.150:60060) 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568a59c640 1 -- 172.21.15.150:0/2906271324 learned_addr learned my addr 172.21.15.150:0/2906271324 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568ad9d640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127186270 0x55a127555010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 0x55a127554ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.898 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568a59c640 1 -- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127186270 msgr2=0x55a127555010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.894+0000 7f568a59c640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127186270 0x55a127555010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f568a59c640 1 -- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 msgr2=0x55a127554ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f568a59c640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 0x55a127554ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f568a59c640 1 -- 172.21.15.150:0/2906271324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a127527eb0 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 0x55a127554ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f568a59c640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f567c00dab0 tx=0x7f567c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9391c8b642101b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c027630 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f567c0277d0 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5684099070 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f567c034050 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 10 client.5961 fetch_fsmap learned FSMap version 30 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x55a127528750 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5684099070 con 0x55a12715d3b0 2024-04-25T14:04:34.899 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7f567c037020 con 0x55a12715d3b0 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 10 client.5961 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 10 client.5961 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 20 client.5961 populate_metadata read hostname 'smithi150' 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a12754f730 0x55a12758ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 4 client.5961 mds_command: new command op to 5832 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.895+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 --> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] -- command(tid 0: {"prefix": "status"}) -- 0x55a127522f60 con 0x55a12754f730 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a12754f730 0x55a12758ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f56697fa640 10 client.5961 ms_handle_connect on v2:172.21.15.187:6833/133937444 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5689d9b640 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a12754f730 0x55a12758ebc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d34eddbf70488118 server_cookie=b7776fd426c2da4f in_seq=0 out_seq=0 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f56697fa640 1 -- 172.21.15.150:0/2906271324 <== mds.0 v2:172.21.15.187:6833/133937444 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55a127522f60 con 0x55a12754f730 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f56697fa640 10 client.5961 handle_command_reply: tid=0 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5697e7d740 1 client.5961 shutdown 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 msgr2=0x55a12757eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.900 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f567c00dab0 tx=0x7f567c004a40 comp rx=0 tx=0).stop 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 shutdown_connections 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.896+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x55a12754f730 0x55a12758ebc0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55a127186270 0x55a127555010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f56697fa640 0 client.5961 ms_handle_reset on v2:172.21.15.187:6833/133937444 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55a127165590 0x55a127554ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 --2- 172.21.15.150:0/2906271324 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55a12715d3b0 0x55a12757eca0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 >> 172.21.15.150:0/2906271324 conn(0x55a12717a3c0 msgr2=0x55a12758fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 shutdown_connections 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 1 -- 172.21.15.150:0/2906271324 wait complete. 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5697e7d740 20 client.5961 trim_cache size 0 max 16384 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:04:34.901 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5832, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 4.7016325280000002, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 30, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 56, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 56, 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 10.023245276999999 2024-04-25T14:04:34.902 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:04:34.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 msgr2=0x7f566c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 0x7f566c0769c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f567c00f500 tx=0x7f567c007770 comp rx=0 tx=0).stop 2024-04-25T14:04:34.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 msgr2=0x7f568c0acd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:34.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f568402a240 tx=0x7f56840046c0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.903 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 shutdown_connections 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f566c0744b0 0x7f566c0769c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f568c0b91f0 0x7f568c0acd90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f568c0b4c10 0x7f568c0ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 --2- 172.21.15.150:0/2978975904 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f568c06d800 0x7f568c0cfe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 >> 172.21.15.150:0/2978975904 conn(0x7f568c0b7690 msgr2=0x7f568c0bb420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 shutdown_connections 2024-04-25T14:04:34.904 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:34.897+0000 7f5690fce640 1 -- 172.21.15.150:0/2978975904 wait complete. 2024-04-25T14:04:34.908 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-25T14:04:35.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.016+0000 7f5592414640 1 Processor -- start 2024-04-25T14:04:35.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.016+0000 7f5592414640 1 -- start start 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f5592414640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c0b5b10 0x7f558c1625f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f5592414640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f558c15e520 0x7f558c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f5592414640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f558c15ee70 con 0x7f558c158760 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f5592414640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f558c070fd0 con 0x7f558c15e520 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f5592414640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f558c15f100 con 0x7f558c0b5b10 2024-04-25T14:04:35.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f558bfff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f558c158760 0x7f558c0b5400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41222/0 (socket says 172.21.15.150:41222) 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f558bfff640 1 -- 172.21.15.150:0/1878365749 learned_addr learned my addr 172.21.15.150:0/1878365749 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f558b7fe640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c0b5b10 0x7f558c1625f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f55909b1640 1 --2- 172.21.15.150:0/1878365749 >> v2:172.21.15.187:3300/0 conn(0x7f558c15e520 0x7f558c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.017+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3655772252 0 0) 0x7f558c15ee70 con 0x7f558c158760 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5574003520 con 0x7f558c158760 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2038222741 0 0) 0x7f5574003520 con 0x7f558c158760 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f55909b1640 1 -- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c0b5b10 msgr2=0x7f558c1625f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f55909b1640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c0b5b10 0x7f558c1625f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f55909b1640 1 -- 172.21.15.150:0/1878365749 >> v1:172.21.15.150:6789/0 conn(0x7f558c158760 legacy=0x7f558c0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.018+0000 7f55909b1640 1 -- 172.21.15.150:0/1878365749 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c15f3e0 con 0x7f558c15e520 2024-04-25T14:04:35.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f55909b1640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c15e930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5584009d90 tx=0x7f558402a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92be0e0fac24610b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:35.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c001670 con 0x7f558c15e520 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5584007540 con 0x7f558c15e520 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f558affd640 1 -- 172.21.15.150:0/1878365749 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55840078f0 con 0x7f558c15e520 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 -- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 msgr2=0x7f558c15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c15e930 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5584009d90 tx=0x7f558402a5f0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 -- 172.21.15.150:0/1878365749 shutdown_connections 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c15e930 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 --2- 172.21.15.150:0/1878365749 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c0b5b10 0x7f558c1625f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.019+0000 7f5592414640 1 -- 172.21.15.150:0/1878365749 >> 172.21.15.150:0/1878365749 conn(0x7f558c0d0400 msgr2=0x7f558c0d0810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.020+0000 7f5592414640 1 -- 172.21.15.150:0/1878365749 shutdown_connections 2024-04-25T14:04:35.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.020+0000 7f5592414640 1 -- 172.21.15.150:0/1878365749 wait complete. 2024-04-25T14:04:35.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.021+0000 7f5592414640 1 Processor -- start 2024-04-25T14:04:35.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.021+0000 7f5592414640 1 -- start start 2024-04-25T14:04:35.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c1f5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c1fa530 0x7f558c0ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f558c1620d0 con 0x7f558c158760 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f558c161df0 con 0x7f558c15e520 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f558c0703d0 con 0x7f558c1fa530 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60064/0 (socket says 172.21.15.150:60064) 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 -- 172.21.15.150:0/3043428197 learned_addr learned my addr 172.21.15.150:0/3043428197 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 -- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c1fa530 msgr2=0x7f558c0ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c1fa530 0x7f558c0ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 -- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 msgr2=0x7f558c1f5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558b7fe640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c1f5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c1f5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 -- 172.21.15.150:0/3043428197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f558c15a380 con 0x7f558c158760 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558b7fe640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c1f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f558bfff640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c5b0 tx=0x7f557c00ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c017070 con 0x7f558c158760 2024-04-25T14:04:35.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f557c005ce0 con 0x7f558c158760 2024-04-25T14:04:35.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f557c007330 con 0x7f558c158760 2024-04-25T14:04:35.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.023+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f558c15a660 con 0x7f558c158760 2024-04-25T14:04:35.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.024+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f558c06e3f0 con 0x7f558c158760 2024-04-25T14:04:35.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.024+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f558c076a40 con 0x7f558c158760 2024-04-25T14:04:35.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.025+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f557c01a070 con 0x7f558c158760 2024-04-25T14:04:35.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.025+0000 7f5588ff9640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 0x7f55780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:35.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.026+0000 7f558b7fe640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 0x7f55780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:35.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.026+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f557c094740 con 0x7f558c158760 2024-04-25T14:04:35.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.026+0000 7f558b7fe640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 0x7f55780769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f558c15e240 tx=0x7f5580008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:35.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.030+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f557c05e790 con 0x7f558c158760 2024-04-25T14:04:35.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.247+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f558c0d1c80 con 0x7f558c158760 2024-04-25T14:04:35.246 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.247+0000 7f5588ff9640 1 -- 172.21.15.150:0/3043428197 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v30) ==== 53+0+73 (secure 0 0 0) 0x7f557c062a90 con 0x7f558c158760 2024-04-25T14:04:35.247 INFO:teuthology.orchestra.run.smithi150.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 msgr2=0x7f55780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 0x7f55780769c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f558c15e240 tx=0x7f5580008040 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 msgr2=0x7f558c0cc310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f557c00c5b0 tx=0x7f557c00ca70 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 shutdown_connections 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f55780744b0 0x7f55780769c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f558c1fa530 0x7f558c0ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f558c15e520 0x7f558c1f5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 --2- 172.21.15.150:0/3043428197 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f558c158760 0x7f558c0cc310 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 >> 172.21.15.150:0/3043428197 conn(0x7f558c0d0400 msgr2=0x7f558c0b6e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 shutdown_connections 2024-04-25T14:04:35.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35.249+0000 7f5592414640 1 -- 172.21.15.150:0/3043428197 wait complete. 2024-04-25T14:04:35.259 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-25T14:04:35.259 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi150.front.sepia.ceph.com 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T14:04:35.260 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:04:35.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.260 DEBUG:teuthology.orchestra.run.smithi150:> ip addr 2024-04-25T14:04:35.275 INFO:teuthology.orchestra.run.smithi150.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T14:04:35.275 INFO:teuthology.orchestra.run.smithi150.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 ::1/128 scope host 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bc brd ff:ff:ff:ff:ff:ff 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f0 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:d9:da:bd brd ff:ff:ff:ff:ff:ff 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: altname enp7s0f1 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:34 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 0c:c4:7a:88:6e:35 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:04:35.276 INFO:teuthology.orchestra.run.smithi150.stdout: inet 172.21.15.150/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft 42625sec preferred_lft 42625sec 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::eb5b:7bab:e07b:2aac/64 scope link noprefixroute 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 6a:dc:79:80:07:d5 brd ff:ff:ff:ff:ff:ff 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::e835:8ff:febc:ce5a/64 scope link 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether fe:50:c0:ba:66:ff brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::fc50:c0ff:feba:66ff/64 scope link 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-25T14:04:35.277 INFO:teuthology.orchestra.run.smithi150.stdout: link/ether 6a:dc:79:80:07:d5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T14:04:35.278 INFO:teuthology.orchestra.run.smithi150.stdout: inet6 fe80::68dc:79ff:fe80:7d5/64 scope link 2024-04-25T14:04:35.278 INFO:teuthology.orchestra.run.smithi150.stdout: valid_lft forever preferred_lft forever 2024-04-25T14:04:35.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.278 DEBUG:teuthology.orchestra.run.smithi150:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T14:04:35.338 INFO:teuthology.orchestra.run.smithi150.stdout:1 2024-04-25T14:04:35.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.340 DEBUG:teuthology.orchestra.run.smithi150:> route 2024-04-25T14:04:35.395 INFO:teuthology.orchestra.run.smithi150.stdout:Kernel IP routing table 2024-04-25T14:04:35.395 INFO:teuthology.orchestra.run.smithi150.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T14:04:35.395 INFO:teuthology.orchestra.run.smithi150.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-25T14:04:35.396 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-25T14:04:35.396 INFO:teuthology.orchestra.run.smithi150.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> set -e 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-25T14:04:35.396 DEBUG:teuthology.orchestra.run.smithi150:> ') 2024-04-25T14:04:35.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T14:04:35.513 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:35.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.519 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:04:35.534 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T14:04:35.534 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T14:04:35.534 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:04:35.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.534 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link set brx.0 up 2024-04-25T14:04:35.597 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:04:35.598 DEBUG:teuthology.orchestra.run.smithi150:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T14:04:35.652 INFO:teuthology.orchestra.run.smithi150.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T14:04:35.653 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T14:04:35.653 DEBUG:teuthology.orchestra.run.smithi150:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:04:35.707 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/ 2024-04-25T14:04:35.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:04:35.799 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:04:35.799 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:04:35.808 INFO:teuthology.orchestra.run.smithi150.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback 2024-04-25T14:04:35.808 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes". 2024-04-25T14:04:35.808 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: resolved to: "172.21.15.150:3300,172.21.15.187:3300,172.21.15.187:3301" 2024-04-25T14:04:35.808 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: trying mount with new device syntax: 0@77163e42-55ae-4cab-b506-bd8c531cc087.cephfs=/ 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,ms_mode=prefer-crc,key=0,mon_addr=172.21.15.150:3300/172.21.15.187:3300/172.21.15.187:3301" will pass to kernel 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout: You just mounted a file system that supports labels which does not 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T14:04:35.809 INFO:teuthology.orchestra.run.smithi150.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T14:04:35.811 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T14:04:35.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:04:35.811 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T14:04:35.834 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T14:04:35.853 INFO:tasks.ceph.mds.d.smithi187.stderr:2024-04-25T14:04:35.853+0000 7f2297530640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:04:35.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T14:04:35.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:04:35.936 INFO:teuthology.orchestra.run.smithi150.stdout:client5285 2024-04-25T14:04:35.937 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:04:35.937 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:04:35.953 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:04:35.994 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-25T14:04:36.033 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.034 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.035 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-25T14:04:36.036 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.037 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-25T14:04:36.038 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:04:36.039 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-25T14:04:36.040 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-04-25T14:04:36.041 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-04-25T14:04:36.042 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:04:36.043 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:04:36.044 INFO:teuthology.orchestra.run.smithi150.stdout: osd_deep_scrub_large_omap_object_key_threshold = 1 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-25T14:04:36.045 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-25T14:04:36.046 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-25T14:04:36.047 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:04:36.048 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-25T14:04:36.049 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-25T14:04:36.050 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:04:36.051 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:04:36.052 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-25T14:04:36.053 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:04:36.054 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:04:36.055 INFO:teuthology.orchestra.run.smithi187.stdout: osd_deep_scrub_large_omap_object_key_threshold = 1 2024-04-25T14:04:36.057 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-25T14:04:36.057 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-25T14:04:36.057 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T14:04:36.057 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:36.057 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-25T14:04:36.058 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-25T14:04:36.058 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T14:04:36.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:36.058 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-25T14:04:36.058 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-25T14:04:36.058 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T14:04:36.058 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:36.059 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-25T14:04:36.059 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-25T14:04:36.059 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T14:04:36.059 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:04:36.108 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T14:04:36.108 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-25T14:04:36.109 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T14:04:36.109 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-25T14:04:36.110 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T14:04:36.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-25T14:04:36.111 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T14:04:36.111 DEBUG:teuthology.orchestra.run.smithi187:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-25T14:04:36.150 INFO:tasks.ceph.mds.a:Started 2024-04-25T14:04:36.151 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:36.152 INFO:tasks.ceph.mds.c:Started 2024-04-25T14:04:36.152 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:36.153 INFO:tasks.ceph.mds.b:Started 2024-04-25T14:04:36.153 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:36.153 INFO:tasks.ceph.mds.d:Started 2024-04-25T14:04:36.154 DEBUG:teuthology.parallel:result is None 2024-04-25T14:04:36.154 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:04:36.236 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:04:36.237 INFO:tasks.ceph.mds.c.smithi150.stdout:starting mds.c at 2024-04-25T14:04:36.241 INFO:tasks.ceph.mds.b.smithi187.stdout:starting mds.b at 2024-04-25T14:04:36.241 INFO:tasks.ceph.mds.d.smithi187.stdout:starting mds.d at 2024-04-25T14:04:36.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.270+0000 7f317fb34640 1 Processor -- start 2024-04-25T14:04:36.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.271+0000 7f317fb34640 1 -- start start 2024-04-25T14:04:36.272 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317fb34640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f317815e050 0x7f31780b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317fb34640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31780b5e70 con 0x7f31780b5a60 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317fb34640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31780cebe0 con 0x7f3178161bd0 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317fb34640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f31780d2080 con 0x7f317815e050 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317e0d1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f31780b5a60 0x7f31780d27f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41236/0 (socket says 172.21.15.150:41236) 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317e0d1640 1 -- 172.21.15.150:0/1370526132 learned_addr learned my addr 172.21.15.150:0/1370526132 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317d0cf640 1 --2- 172.21.15.150:0/1370526132 >> v2:172.21.15.187:3300/0 conn(0x7f317815e050 0x7f31780b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4029991155 0 0) 0x7f3170001110 con 0x7f31780b5a60 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.272+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3164003520 con 0x7f31780b5a60 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3848109260 0 0) 0x7f31780cebe0 con 0x7f3178161bd0 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3170001110 con 0x7f3178161bd0 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317d0cf640 1 -- 172.21.15.150:0/1370526132 >> v1:172.21.15.187:6789/0 conn(0x7f3178161bd0 legacy=0x7f31780ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317d0cf640 1 -- 172.21.15.150:0/1370526132 >> v1:172.21.15.150:6789/0 conn(0x7f31780b5a60 legacy=0x7f31780d27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317d0cf640 1 -- 172.21.15.150:0/1370526132 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31780ceec0 con 0x7f317815e050 2024-04-25T14:04:36.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.273+0000 7f317d0cf640 1 --2- 172.21.15.150:0/1370526132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780b5520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31680097a0 tx=0x7f316802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61db5921880594ce server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:36.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168037070 con 0x7f317815e050 2024-04-25T14:04:36.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3168032440 con 0x7f317815e050 2024-04-25T14:04:36.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317c8ce640 1 -- 172.21.15.150:0/1370526132 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3168032810 con 0x7f317815e050 2024-04-25T14:04:36.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 -- 172.21.15.150:0/1370526132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 msgr2=0x7f31780b5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 --2- 172.21.15.150:0/1370526132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780b5520 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31680097a0 tx=0x7f316802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 -- 172.21.15.150:0/1370526132 shutdown_connections 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 --2- 172.21.15.150:0/1370526132 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780b5520 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 -- 172.21.15.150:0/1370526132 >> 172.21.15.150:0/1370526132 conn(0x7f31780d0520 msgr2=0x7f31780d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 -- 172.21.15.150:0/1370526132 shutdown_connections 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.274+0000 7f317fb34640 1 -- 172.21.15.150:0/1370526132 wait complete. 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.276+0000 7f317fb34640 1 Processor -- start 2024-04-25T14:04:36.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.276+0000 7f317fb34640 1 -- start start 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31780b5a60 0x7f3178150100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f317815e460 con 0x7f3178161bd0 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f31780d22d0 con 0x7f317815e050 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317fb34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f31780b6d80 con 0x7f31780b5a60 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.277+0000 7f317e0d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60106/0 (socket says 172.21.15.150:60106) 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 -- 172.21.15.150:0/174572166 learned_addr learned my addr 172.21.15.150:0/174572166 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317d0cf640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:36.278 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317d8d0640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31780b5a60 0x7f3178150100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 -- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31780b5a60 msgr2=0x7f3178150100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31780b5a60 0x7f3178150100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 -- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 msgr2=0x7f31780c9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780c9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 -- 172.21.15.150:0/174572166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31780c9970 con 0x7f3178161bd0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317e0d1640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f317000c730 tx=0x7f317000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170017070 con 0x7f3178161bd0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3170005ad0 con 0x7f3178161bd0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31781f7c20 con 0x7f3178161bd0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3170007070 con 0x7f3178161bd0 2024-04-25T14:04:36.279 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.278+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31781f81e0 con 0x7f3178161bd0 2024-04-25T14:04:36.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.279+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31780c96e0 con 0x7f3178161bd0 2024-04-25T14:04:36.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.280+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f317001a070 con 0x7f3178161bd0 2024-04-25T14:04:36.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.280+0000 7f316e7fc640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 0x7f315c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:36.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.280+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7f3170094840 con 0x7f3178161bd0 2024-04-25T14:04:36.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.280+0000 7f317d8d0640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 0x7f315c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:36.281 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.281+0000 7f317d8d0640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 0x7f315c0769c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3174001560 tx=0x7f3174009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:36.284 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.285+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f31780b5e70 con 0x7f3178161bd0 2024-04-25T14:04:36.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.494+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f31780c9c50 con 0x7f3178161bd0 2024-04-25T14:04:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.495+0000 7f316e7fc640 1 -- 172.21.15.150:0/174572166 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+5187 (secure 0 0 0) 0x7f317005eed0 con 0x7f3178161bd0 2024-04-25T14:04:36.494 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:04:36.494 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":30,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/468558883","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":468558883},{"type":"v1","addr":"172.21.15.187:6834","nonce":468558883}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5820,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6835/3011235888","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":3011235888},{"type":"v1","addr":"172.21.15.150:6835","nonce":3011235888}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27},{"gid":5823,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/1873279280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6836","nonce":1873279280},{"type":"v1","addr":"172.21.15.150:6837","nonce":1873279280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:31.564985+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5832},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5832":{"gid":5832,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.187:6835/133937444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":133937444},{"type":"v1","addr":"172.21.15.187:6835","nonce":133937444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5832,"qdb_cluster":[5832]},"id":3}]} 2024-04-25T14:04:36.494 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 30 2024-04-25T14:04:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 msgr2=0x7f315c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 0x7f315c0769c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3174001560 tx=0x7f3174009040 comp rx=0 tx=0).stop 2024-04-25T14:04:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 msgr2=0x7f31780ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f317000c730 tx=0x7f317000cbf0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 shutdown_connections 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f315c0744b0 0x7f315c0769c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f3178161bd0 0x7f31780ccce0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f317815e050 0x7f31780c9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 --2- 172.21.15.150:0/174572166 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f31780b5a60 0x7f3178150100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 >> 172.21.15.150:0/174572166 conn(0x7f31780d0520 msgr2=0x7f31780d3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 shutdown_connections 2024-04-25T14:04:36.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:36.497+0000 7f317fb34640 1 -- 172.21.15.150:0/174572166 wait complete. 2024-04-25T14:04:36.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 30, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:04:29.564574+0000', 'modified': '2024-04-25T14:04:31.564985+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5832}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5832': {'gid': 5832, 'name': 'd', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.187:6835/133937444', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.187:6833', 'nonce': 133937444}, {'type': 'v1', 'addr': '172.21.15.187:6835', 'nonce': 133937444}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5832, 'qdb_cluster': [5832]} 2024-04-25T14:04:36.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:04:36.910 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-25T14:04:37.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.021+0000 7fbc0e957640 1 Processor -- start 2024-04-25T14:04:37.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.021+0000 7fbc0e957640 1 -- start start 2024-04-25T14:04:37.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0e957640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 0x7fbc0815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0e957640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc080b5c30 0x7fbc080d27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0e957640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc080cebe0 con 0x7fbc080ce390 2024-04-25T14:04:37.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0e957640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbc080d2080 con 0x7fbc080b5c30 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0e957640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbc080ceec0 con 0x7fbc08161bd0 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0cef4640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbc080b5c30 0x7fbc080d27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc077fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbc080ce390 0x7fbc080b5520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41248/0 (socket says 172.21.15.150:41248) 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc077fe640 1 -- 172.21.15.150:0/997014099 learned_addr learned my addr 172.21.15.150:0/997014099 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc07fff640 1 --2- 172.21.15.150:0/997014099 >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 0x7fbc0815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0cef4640 1 -- 172.21.15.150:0/997014099 >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 msgr2=0x7fbc0815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/997014099 >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 0x7fbc0815e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0cef4640 1 -- 172.21.15.150:0/997014099 >> v1:172.21.15.150:6789/0 conn(0x7fbc080ce390 legacy=0x7fbc080b5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc0cef4640 1 -- 172.21.15.150:0/997014099 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc080cf1a0 con 0x7fbc080b5c30 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc06ffd640 1 -- 172.21.15.150:0/997014099 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231516220 0 0) 0x7fbc080cebe0 con 0x7fbc080ce390 2024-04-25T14:04:37.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.023+0000 7fbc07fff640 1 --2- 172.21.15.150:0/997014099 >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 0x7fbc0815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.024+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/997014099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080b5c30 0x7fbc080d27f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbc080b6310 tx=0x7fbc0002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c43ebe36d57f8e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.024+0000 7fbc06ffd640 1 -- 172.21.15.150:0/997014099 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00007740 con 0x7fbc080b5c30 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.024+0000 7fbc06ffd640 1 -- 172.21.15.150:0/997014099 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc000078e0 con 0x7fbc080b5c30 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.024+0000 7fbc06ffd640 1 -- 172.21.15.150:0/997014099 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00007cb0 con 0x7fbc080b5c30 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 -- 172.21.15.150:0/997014099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080b5c30 msgr2=0x7fbc080d27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 --2- 172.21.15.150:0/997014099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080b5c30 0x7fbc080d27f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fbc080b6310 tx=0x7fbc0002a5f0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 -- 172.21.15.150:0/997014099 shutdown_connections 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 --2- 172.21.15.150:0/997014099 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080b5c30 0x7fbc080d27f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 --2- 172.21.15.150:0/997014099 >> v2:172.21.15.187:3300/0 conn(0x7fbc08161bd0 0x7fbc0815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 -- 172.21.15.150:0/997014099 >> 172.21.15.150:0/997014099 conn(0x7fbc080d0520 msgr2=0x7fbc080d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 -- 172.21.15.150:0/997014099 shutdown_connections 2024-04-25T14:04:37.025 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.025+0000 7fbc0e957640 1 -- 172.21.15.150:0/997014099 wait complete. 2024-04-25T14:04:37.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.027+0000 7fbc0e957640 1 Processor -- start 2024-04-25T14:04:37.026 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.027+0000 7fbc0e957640 1 -- start start 2024-04-25T14:04:37.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080ce390 0x7fbc080cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc08161bd0 0x7fbc08158fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc080cd740 0x7fbc080ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbc080b6f00 con 0x7fbc080ce390 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbc080b6d80 con 0x7fbc08161bd0 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0e957640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbc080ce7a0 con 0x7fbc080cd740 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080ce390 0x7fbc080cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080ce390 0x7fbc080cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60124/0 (socket says 172.21.15.150:60124) 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 -- 172.21.15.150:0/2617027740 learned_addr learned my addr 172.21.15.150:0/2617027740 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc08161bd0 0x7fbc08158fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc080cd740 0x7fbc080ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 -- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc080cd740 msgr2=0x7fbc080ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc080cd740 0x7fbc080ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 -- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc08161bd0 msgr2=0x7fbc08158fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc08161bd0 0x7fbc08158fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc07fff640 1 -- 172.21.15.150:0/2617027740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc080c92b0 con 0x7fbc080ce390 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbc080cd740 0x7fbc080ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.028+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbc08161bd0 0x7fbc08158fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbc080ce390 0x7fbc080cb060 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fbbf800c7f0 tx=0x7fbbf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc04ff9640 1 -- 172.21.15.150:0/2617027740 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8015070 con 0x7fbc080ce390 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc04ff9640 1 -- 172.21.15.150:0/2617027740 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8007820 con 0x7fbc080ce390 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc0e957640 1 -- 172.21.15.150:0/2617027740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc080c9590 con 0x7fbc080ce390 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc04ff9640 1 -- 172.21.15.150:0/2617027740 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8007c10 con 0x7fbc080ce390 2024-04-25T14:04:37.030 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.029+0000 7fbc0e957640 1 -- 172.21.15.150:0/2617027740 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc080c9b50 con 0x7fbc080ce390 2024-04-25T14:04:37.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.031+0000 7fbc04ff9640 1 -- 172.21.15.150:0/2617027740 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbbf8018070 con 0x7fbc080ce390 2024-04-25T14:04:37.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.031+0000 7fbc04ff9640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbbf40744e0 0x7fbbf40769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.031+0000 7fbc04ff9640 1 -- 172.21.15.150:0/2617027740 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fbbf80950f0 con 0x7fbc080ce390 2024-04-25T14:04:37.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.031+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbbf40744e0 0x7fbbf40769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.031 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.031+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2617027740 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbbf40744e0 0x7fbbf40769f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fbc080b2da0 tx=0x7fbbfc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:37.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.032+0000 7fbc15806740 1 Processor -- start 2024-04-25T14:04:37.032 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.032+0000 7fbc15806740 1 -- start start 2024-04-25T14:04:37.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d75483b0 0x5635d7939fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 0x5635d7913260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635d7877740 con 0x5635d7571270 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5635d78133a0 con 0x5635d7550590 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc15806740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5635d7873700 con 0x5635d75483b0 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60130/0 (socket says 172.21.15.150:60130) 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 -- 172.21.15.150:0/766113361 learned_addr learned my addr 172.21.15.150:0/766113361 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc077fe640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d75483b0 0x5635d7939fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 0x5635d7913260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 -- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d75483b0 msgr2=0x5635d7939fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d75483b0 0x5635d7939fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 -- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 msgr2=0x5635d7913260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 0x5635d7913260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc07fff640 1 -- 172.21.15.150:0/766113361 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635d782f750 con 0x5635d7571270 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.033+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 0x5635d7913260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:04:37.035 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbc07fff640 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbbf805b100 tx=0x7fbbf805ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b983b4df05a330f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbbedffb640 1 -- 172.21.15.150:0/766113361 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf801f040 con 0x5635d7571270 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbbedffb640 1 -- 172.21.15.150:0/766113361 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8063430 con 0x5635d7571270 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbbedffb640 1 -- 172.21.15.150:0/766113361 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8063220 con 0x5635d7571270 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbc15806740 1 -- 172.21.15.150:0/766113361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 msgr2=0x5635d793d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.034+0000 7fbc15806740 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbbf805b100 tx=0x7fbbf805ecd0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 -- 172.21.15.150:0/766113361 shutdown_connections 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7550590 0x5635d7913260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d75483b0 0x5635d7939fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 --2- 172.21.15.150:0/766113361 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d7571270 0x5635d793d600 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 -- 172.21.15.150:0/766113361 >> 172.21.15.150:0/766113361 conn(0x5635d75653c0 msgr2=0x5635d79129f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 -- 172.21.15.150:0/766113361 shutdown_connections 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 -- 172.21.15.150:0/766113361 wait complete. 2024-04-25T14:04:37.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.035+0000 7fbc15806740 1 -- start start 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.036+0000 7fbc15806740 1 Processor -- start 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d75483b0 0x5635d7957240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d7550590 0x5635d7940c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7571270 0x5635d7955d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x5635d77fefe0 con 0x5635d75483b0 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x5635d7956a80 con 0x5635d7571270 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x5635d796b560 con 0x5635d7550590 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc15806740 10 client.0 _ll_register_callbacks cb 0x5635d7815150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d75483b0 0x5635d7957240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d75483b0 0x5635d7957240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:60134/0 (socket says 172.21.15.150:60134) 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 -- 172.21.15.150:0/2477027080 learned_addr learned my addr 172.21.15.150:0/2477027080 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d7550590 0x5635d7940c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7571270 0x5635d7955d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.040 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d7550590 msgr2=0x5635d7940c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x5635d7550590 0x5635d7940c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7571270 msgr2=0x5635d7955d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7571270 0x5635d7955d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc07fff640 1 -- 172.21.15.150:0/2477027080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635d75892d0 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.037+0000 7fbc0cef4640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x5635d7571270 0x5635d7955d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc07fff640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x5635d75483b0 0x5635d7957240 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fbbf805c2f0 tx=0x7fbbf800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef431d6781763dfa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 1 -- 172.21.15.150:0/2477027080 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf8090070 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 1 -- 172.21.15.150:0/2477027080 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(56..56 src has 1..56) ==== 6101+0+0 (secure 0 0 0) 0x7fbbf805a9f0 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 1 -- 172.21.15.150:0/2477027080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbf8063430 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 1 -- 172.21.15.150:0/2477027080 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbbf801f040 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 10 client.6012 fetch_fsmap learned FSMap version 30 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 1 -- 172.21.15.150:0/2477027080 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=2+}) -- 0x5635d76536b0 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 1 -- 172.21.15.150:0/2477027080 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf8063430 con 0x5635d75483b0 2024-04-25T14:04:37.041 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbbe6ffd640 1 -- 172.21.15.150:0/2477027080 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 30) ==== 3494+0+0 (secure 0 0 0) 0x7fbbf8022060 con 0x5635d75483b0 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 10 client.6012 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 10 client.6012 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 20 client.6012 populate_metadata read hostname 'smithi150' 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 4 client.6012 mds_command: new command op to 5832 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc15806740 1 -- 172.21.15.150:0/2477027080 --> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635d7885350 con 0x5635d752fe40 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.038+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.039+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.039+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.039+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:37.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.039+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T14:04:37.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.240+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.242+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:37.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.242+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:37.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.242+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:37.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.242+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T14:04:37.252 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:04:37.253+0000 7f46d1fd1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:04:37.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.643+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:37.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.644+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:37.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.644+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:37.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.644+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:37.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:37.644+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T14:04:38.444 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:38.445+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:38.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:38.446+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:38.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:38.446+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:38.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:38.446+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:38.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:38.446+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T14:04:38.939 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:04:38.942 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:04:40.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:40.048+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:40.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:40.048+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:40.047 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:40.048+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:40.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:40.048+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:40.048 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:40.048+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T14:04:42.249 INFO:tasks.ceph.mds.a.smithi150.stderr:2024-04-25T14:04:42.249+0000 7f5c27a1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:04:43.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:43.251+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:43.252+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:43.252+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:43.252+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:43.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:43.252+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-25T14:04:49.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:49.658+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:04:49.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:49.658+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:04:49.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:49.658+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:04:49.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:49.658+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:04:49.659 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:04:49.658+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-04-25T14:05:02.463 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:02.463+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:05:02.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:02.464+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:05:02.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:02.464+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:05:02.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:02.464+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:05:02.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:02.464+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:05:09.196 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:05:09.200 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:05:17.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:17.480+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:05:17.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:17.481+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:05:17.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:17.481+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:05:17.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:17.481+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:05:17.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:17.481+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:05:32.481 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:32.482+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:05:32.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:32.482+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:05:32.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:32.482+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:05:32.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:32.482+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:05:32.482 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:32.482+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:05:39.268 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:05:39.272 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:05:47.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:47.497+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:05:47.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:47.497+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:05:47.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:47.497+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:05:47.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:47.497+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:05:47.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:05:47.497+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:06:02.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:02.513+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:02.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:02.514+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:06:02.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:02.514+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:06:02.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:02.514+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:06:02.514 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:02.514+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:06:09.400 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:06:09.403 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:06:17.529 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:17.530+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:17.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:17.531+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:06:17.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:17.531+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:06:17.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:17.531+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:06:17.530 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:17.531+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:06:32.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:32.546+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:32.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:32.547+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 19 2024-04-25T14:06:32.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:32.547+0000 7fbc077fe640 1 -- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 msgr2=0x5635d7936160 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-25T14:06:32.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:32.547+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T14:06:32.546 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:32.547+0000 7fbc077fe640 1 --2- 172.21.15.150:0/2477027080 >> [v2:172.21.15.187:6833/133937444,v1:172.21.15.187:6835/133937444] conn(0x5635d752fe40 0x5635d7936160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-04-25T14:06:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-25T14:06:37.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:38.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.063+0000 7f8e0b4d8640 1 Processor -- start 2024-04-25T14:06:38.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.063+0000 7f8e0b4d8640 1 -- start start 2024-04-25T14:06:38.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e0b4d8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e040b07c0 0x7f8e0415dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e0b4d8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 0x7f8e04073e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e0b4d8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e04074360 con 0x7f8e0415ac80 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e0b4d8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8e0415da10 con 0x7f8e040b07c0 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e0b4d8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8e040b0bd0 con 0x7f8e04073a10 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e040b07c0 0x7f8e0415dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e040b07c0 0x7f8e0415dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43250/0 (socket says 172.21.15.150:43250) 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 -- 172.21.15.150:0/3416376983 learned_addr learned my addr 172.21.15.150:0/3416376983 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e09a75640 1 --2- 172.21.15.150:0/3416376983 >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 0x7f8e04073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 -- 172.21.15.150:0/3416376983 >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 msgr2=0x7f8e04073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 --2- 172.21.15.150:0/3416376983 >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 0x7f8e04073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 -- 172.21.15.150:0/3416376983 >> v1:172.21.15.187:6789/0 conn(0x7f8e0415ac80 legacy=0x7f8e040b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e08a73640 1 -- 172.21.15.150:0/3416376983 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e040b0ea0 con 0x7f8e040b07c0 2024-04-25T14:06:38.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.065+0000 7f8e09a75640 1 --2- 172.21.15.150:0/3416376983 >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 0x7f8e04073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.066+0000 7f8e08a73640 1 --2- 172.21.15.150:0/3416376983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e040b07c0 0x7f8e0415dea0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e04073150 tx=0x7f8df402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af424ba195f1f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.066+0000 7f8dfbfff640 1 -- 172.21.15.150:0/3416376983 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4037070 con 0x7f8e040b07c0 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.066+0000 7f8dfbfff640 1 -- 172.21.15.150:0/3416376983 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df4032440 con 0x7f8e040b07c0 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.066+0000 7f8dfbfff640 1 -- 172.21.15.150:0/3416376983 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df4032810 con 0x7f8e040b07c0 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/3416376983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e040b07c0 msgr2=0x7f8e0415dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/3416376983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e040b07c0 0x7f8e0415dea0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8e04073150 tx=0x7f8df402a230 comp rx=0 tx=0).stop 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/3416376983 shutdown_connections 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/3416376983 >> v2:172.21.15.187:3300/0 conn(0x7f8e04073a10 0x7f8e04073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/3416376983 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e040b07c0 0x7f8e0415dea0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/3416376983 >> 172.21.15.150:0/3416376983 conn(0x7f8e040d1ee0 msgr2=0x7f8e04073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/3416376983 shutdown_connections 2024-04-25T14:06:38.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.067+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/3416376983 wait complete. 2024-04-25T14:06:38.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.069+0000 7f8e0b4d8640 1 Processor -- start 2024-04-25T14:06:38.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.069+0000 7f8e0b4d8640 1 -- start start 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e04073a10 0x7f8e0409b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e040b07c0 0x7f8e0414a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0415e3e0 con 0x7f8e0415ac80 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0415d6a0 con 0x7f8e040b07c0 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e0b4d8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e040b4c30 con 0x7f8e04073a10 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e09a75640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e09a75640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43258/0 (socket says 172.21.15.150:43258) 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e09a75640 1 -- 172.21.15.150:0/4058154403 learned_addr learned my addr 172.21.15.150:0/4058154403 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e08a73640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e040b07c0 0x7f8e0414a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.070+0000 7f8e09274640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e04073a10 0x7f8e0409b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 -- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e04073a10 msgr2=0x7f8e0409b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.072 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e04073a10 0x7f8e0409b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 -- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e040b07c0 msgr2=0x7f8e0414a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e040b07c0 0x7f8e0414a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 -- 172.21.15.150:0/4058154403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0409d3e0 con 0x7f8e0415ac80 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e09a75640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c5c0 tx=0x7f8dfc00ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc015070 con 0x7f8e0415ac80 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dfc011820 con 0x7f8e0415ac80 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0409c1a0 con 0x7f8e0415ac80 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc011c10 con 0x7f8e0415ac80 2024-04-25T14:06:38.073 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.071+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e041556a0 con 0x7f8e0415ac80 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.072+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0409bf10 con 0x7f8e0415ac80 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.073+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8dfc018070 con 0x7f8e0415ac80 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.073+0000 7f8df9ffb640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 0x7f8de80769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.073+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f8dfc0952b0 con 0x7f8e0415ac80 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.073+0000 7f8e09274640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 0x7f8de80769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.074 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.074+0000 7f8e09274640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 0x7f8de80769f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009f20 tx=0x7f8e000094d0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.078+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8dfc05f820 con 0x7f8e0415ac80 2024-04-25T14:06:38.291 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.291+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8e04050740 con 0x7f8e0415ac80 2024-04-25T14:06:38.292 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.292+0000 7f8df9ffb640 1 -- 172.21.15.150:0/4058154403 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f8dfc05f120 con 0x7f8e0415ac80 2024-04-25T14:06:38.292 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:38.292 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:38.292 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:38.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 msgr2=0x7f8de80769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 0x7f8de80769f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8e00009f20 tx=0x7f8e000094d0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 msgr2=0x7f8e0406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00c5c0 tx=0x7f8dfc00ca80 comp rx=0 tx=0).stop 2024-04-25T14:06:38.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 shutdown_connections 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8de80744e0 0x7f8de80769f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e0415ac80 0x7f8e0406fd00 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e040b07c0 0x7f8e0414a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 --2- 172.21.15.150:0/4058154403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e04073a10 0x7f8e0409b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 >> 172.21.15.150:0/4058154403 conn(0x7f8e040d1ee0 msgr2=0x7f8e040b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 shutdown_connections 2024-04-25T14:06:38.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.294+0000 7f8e0b4d8640 1 -- 172.21.15.150:0/4058154403 wait complete. 2024-04-25T14:06:38.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 37, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-25T14:04:29.564574+0000', 'modified': '2024-04-25T14:04:41.277594+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 59, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5982}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5982': {'gid': 5982, 'name': 'a', 'rank': 0, 'incarnation': 33, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.150:6836/685573768', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.150:6834', 'nonce': 685573768}, {'type': 'v1', 'addr': '172.21.15.150:6836', 'nonce': 685573768}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5982, 'qdb_cluster': [5982]} 2024-04-25T14:06:38.708 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-25T14:06:38.708 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T14:06:38.820 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.820+0000 7f06e5f00640 1 Processor -- start 2024-04-25T14:06:38.821 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.820+0000 7f06e5f00640 1 -- start start 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06e5f00640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06e00cff20 0x7f06e00d28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06e5f00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e015fe40 0x7f06e01626e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06e5f00640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e00d2f60 con 0x7f06e01622d0 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06e5f00640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f06e00d0330 con 0x7f06e00cff20 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06e5f00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e00d3240 con 0x7f06e015fe40 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06e00cff20 0x7f06e00d28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f06e00cff20 0x7f06e00d28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43270/0 (socket says 172.21.15.150:43270) 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 -- 172.21.15.150:0/230414804 learned_addr learned my addr 172.21.15.150:0/230414804 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06dffff640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e015fe40 0x7f06e01626e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 -- 172.21.15.150:0/230414804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e015fe40 msgr2=0x7f06e01626e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e015fe40 0x7f06e01626e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 -- 172.21.15.150:0/230414804 >> v1:172.21.15.187:6789/0 conn(0x7f06e01622d0 legacy=0x7f06e015f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.822+0000 7f06df7fe640 1 -- 172.21.15.150:0/230414804 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e00d3520 con 0x7f06e00cff20 2024-04-25T14:06:38.824 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.823+0000 7f06df7fe640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e00d28d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009850 tx=0x7f06d002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc76f497d990510a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.823+0000 7f06de7fc640 1 -- 172.21.15.150:0/230414804 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0004030 con 0x7f06e00cff20 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.823+0000 7f06de7fc640 1 -- 172.21.15.150:0/230414804 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d00041d0 con 0x7f06e00cff20 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.823+0000 7f06de7fc640 1 -- 172.21.15.150:0/230414804 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0003610 con 0x7f06e00cff20 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 -- 172.21.15.150:0/230414804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 msgr2=0x7f06e00d28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e00d28d0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f06d0009850 tx=0x7f06d002a230 comp rx=0 tx=0).stop 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 -- 172.21.15.150:0/230414804 shutdown_connections 2024-04-25T14:06:38.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e015fe40 0x7f06e01626e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 --2- 172.21.15.150:0/230414804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e00d28d0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 -- 172.21.15.150:0/230414804 >> 172.21.15.150:0/230414804 conn(0x7f06e00b76d0 msgr2=0x7f06e00b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 -- 172.21.15.150:0/230414804 shutdown_connections 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.824+0000 7f06e5f00640 1 -- 172.21.15.150:0/230414804 wait complete. 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.826+0000 7f06e5f00640 1 Processor -- start 2024-04-25T14:06:38.827 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.826+0000 7f06e5f00640 1 -- start start 2024-04-25T14:06:38.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e015fe40 0x7f06e00c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e01622d0 0x7f06e006dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06e00cf190 con 0x7f06e00cff20 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06e00cf010 con 0x7f06e015fe40 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06e5f00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e00b44f0 con 0x7f06e01622d0 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43286/0 (socket says 172.21.15.150:43286) 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06df7fe640 1 -- 172.21.15.150:0/986382437 learned_addr learned my addr 172.21.15.150:0/986382437 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06deffd640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e015fe40 0x7f06e00c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.827+0000 7f06dffff640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e01622d0 0x7f06e006dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 -- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e01622d0 msgr2=0x7f06e006dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e01622d0 0x7f06e006dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 -- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e015fe40 msgr2=0x7f06e00c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e015fe40 0x7f06e00c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 -- 172.21.15.150:0/986382437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e0159e10 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06df7fe640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f06d002a270 tx=0x7f06d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06c7fff640 1 -- 172.21.15.150:0/986382437 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0041020 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06c7fff640 1 -- 172.21.15.150:0/986382437 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d0003c50 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e015a0f0 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06c7fff640 1 -- 172.21.15.150:0/986382437 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d0004030 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.828+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e01598a0 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.830+0000 7f06c7fff640 1 -- 172.21.15.150:0/986382437 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f06d0044020 con 0x7f06e00cff20 2024-04-25T14:06:38.832 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.830+0000 7f06c7fff640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 0x7f06cc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.830+0000 7f06c7fff640 1 -- 172.21.15.150:0/986382437 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f06d00b5fe0 con 0x7f06e00cff20 2024-04-25T14:06:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.830+0000 7f06deffd640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 0x7f06cc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.831+0000 7f06deffd640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 0x7f06cc0769c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f06d4007fd0 tx=0x7f06d4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.841 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.831+0000 7f06ecdb4740 1 Processor -- start 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.831+0000 7f06ecdb4740 1 -- start start 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.832+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bb2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81b8c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fb81af0740 con 0x55fb817ea270 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55fb81a8c3a0 con 0x55fb817c13b0 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fb81aec700 con 0x55fb817c9590 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.842 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43290/0 (socket says 172.21.15.150:43290) 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 -- 172.21.15.150:0/919937203 learned_addr learned my addr 172.21.15.150:0/919937203 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06dffff640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81b8c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06deffd640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bb2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 -- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 msgr2=0x55fb81b8c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81b8c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 -- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 msgr2=0x55fb81bb2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bb2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 -- 172.21.15.150:0/919937203 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb81aa8750 con 0x55fb817ea270 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06deffd640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bb2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:38.843 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06dffff640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81b8c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06df7fe640 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f06d007c4d0 tx=0x7f06d007f8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700b031c85c836e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06c57fa640 1 -- 172.21.15.150:0/919937203 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d007fee0 con 0x55fb817ea270 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06c57fa640 1 -- 172.21.15.150:0/919937203 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d0084290 con 0x55fb817ea270 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.833+0000 7f06c57fa640 1 -- 172.21.15.150:0/919937203 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d007f340 con 0x55fb817ea270 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 -- 172.21.15.150:0/919937203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 msgr2=0x55fb81bb6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f06d007c4d0 tx=0x7f06d007f8f0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 -- 172.21.15.150:0/919937203 shutdown_connections 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81b8c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.844 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bb2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/919937203 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6600 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 -- 172.21.15.150:0/919937203 >> 172.21.15.150:0/919937203 conn(0x55fb817de3c0 msgr2=0x55fb81b8b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 -- 172.21.15.150:0/919937203 shutdown_connections 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.834+0000 7f06ecdb4740 1 -- 172.21.15.150:0/919937203 wait complete. 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.835+0000 7f06ecdb4740 1 -- start start 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.835+0000 7f06ecdb4740 1 Processor -- start 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 0x55fb81bac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.845 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817ea270 0x55fb81bd1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fb81af0740 con 0x55fb817c9590 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55fb81a77fe0 con 0x55fb817ea270 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fb81be1900 con 0x55fb817c13b0 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.836+0000 7f06ecdb4740 10 client.0 _ll_register_callbacks cb 0x55fb81a8e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.846 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43294/0 (socket says 172.21.15.150:43294) 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 -- 172.21.15.150:0/4067304142 learned_addr learned my addr 172.21.15.150:0/4067304142 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06df7fe640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 0x55fb81bac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 -- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 msgr2=0x55fb81bac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06dffff640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817ea270 0x55fb81bd1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 0x55fb81bac4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 -- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817ea270 msgr2=0x55fb81bd1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.847 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817ea270 0x55fb81bd1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 -- 172.21.15.150:0/4067304142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb818022d0 con 0x55fb817c9590 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06df7fe640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 0x55fb81bac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06deffd640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f06d4003870 tx=0x7f06d4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da774dcc3353ef4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d4015940 con 0x55fb817c9590 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f06d4021ca0 con 0x55fb817c9590 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d0084290 con 0x55fb817c9590 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f06d4015ae0 con 0x55fb817c9590 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06ecdb4740 10 client.6030 fetch_fsmap learned FSMap version 37 2024-04-25T14:06:38.848 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.837+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55fb818cc6b0 con 0x55fb817c9590 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d0084290 con 0x55fb817c9590 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f06d4015c80 con 0x55fb817c9590 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 10 client.6030 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 10 client.6030 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 20 client.6030 populate_metadata read hostname 'smithi150' 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb817a8e40 0x55fb81bcf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 4 client.6030 mds_command: new command op to 5982 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 --> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb81afe350 con 0x55fb817a8e40 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06df7fe640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb817a8e40 0x55fb81bcf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06be7fc640 10 client.6030 ms_handle_connect on v2:172.21.15.150:6834/685573768 2024-04-25T14:06:38.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.838+0000 7f06df7fe640 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb817a8e40 0x55fb81bcf170 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f4082a76b82d2cf server_cookie=4c51b31c5e46b09f in_seq=0 out_seq=0 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06be7fc640 1 -- 172.21.15.150:0/4067304142 <== mds.0 v2:172.21.15.150:6834/685573768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fb81afe350 con 0x55fb817a8e40 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06be7fc640 10 client.6030 handle_command_reply: tid=0 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 client.6030 shutdown 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 msgr2=0x55fb81bd00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f06d4003870 tx=0x7f06d4000bc0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 shutdown_connections 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb817a8e40 0x55fb81bcf170 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817ea270 0x55fb81bd1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c9590 0x55fb81bd00e0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.850 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06be7fc640 0 client.6030 ms_handle_reset on v2:172.21.15.150:6834/685573768 2024-04-25T14:06:38.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/4067304142 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c13b0 0x55fb81bac4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 >> 172.21.15.150:0/4067304142 conn(0x55fb817de3c0 msgr2=0x55fb81b8b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 shutdown_connections 2024-04-25T14:06:38.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 1 -- 172.21.15.150:0/4067304142 wait complete. 2024-04-25T14:06:38.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.839+0000 7f06ecdb4740 20 client.6030 trim_cache size 0 max 16384 2024-04-25T14:06:38.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.858+0000 7f06ecdb4740 1 Processor -- start 2024-04-25T14:06:38.858 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.858+0000 7f06ecdb4740 1 -- start start 2024-04-25T14:06:38.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bd1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81bb6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0202730 con 0x55fb817ea270 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f06e0201a90 con 0x55fb817c13b0 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.859+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f06e02020e0 con 0x55fb817c9590 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43304/0 (socket says 172.21.15.150:43304) 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 -- 172.21.15.150:0/1465250736 learned_addr learned my addr 172.21.15.150:0/1465250736 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06df7fe640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bd1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06deffd640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81bb6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 -- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 msgr2=0x55fb81bb6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81bb6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 -- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 msgr2=0x55fb81bd1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bd1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 -- 172.21.15.150:0/1465250736 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb81be2a70 con 0x55fb817ea270 2024-04-25T14:06:38.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06df7fe640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bd1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06dffff640 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009d70 tx=0x7f06d8007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1ef7628f8b1e68f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06c4ff9640 1 -- 172.21.15.150:0/1465250736 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d801a020 con 0x55fb817ea270 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06c4ff9640 1 -- 172.21.15.150:0/1465250736 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d8007e10 con 0x55fb817ea270 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.860+0000 7f06c4ff9640 1 -- 172.21.15.150:0/1465250736 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d80168a0 con 0x55fb817ea270 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 -- 172.21.15.150:0/1465250736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 msgr2=0x55fb81bb6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f06d8009d70 tx=0x7f06d8007700 comp rx=0 tx=0).stop 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 -- 172.21.15.150:0/1465250736 shutdown_connections 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817ea270 0x55fb81bb6b40 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817c9590 0x55fb81bb6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/1465250736 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c13b0 0x55fb81bd1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 -- 172.21.15.150:0/1465250736 >> 172.21.15.150:0/1465250736 conn(0x55fb817de3c0 msgr2=0x55fb81b8ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 -- 172.21.15.150:0/1465250736 shutdown_connections 2024-04-25T14:06:38.864 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.861+0000 7f06ecdb4740 1 -- 172.21.15.150:0/1465250736 wait complete. 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.862+0000 7f06ecdb4740 1 -- start start 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.862+0000 7f06ecdb4740 1 Processor -- start 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c9590 0x55fb81bb6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817ea270 0x55fb81bd8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x55fb81bd0620 con 0x55fb817c13b0 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x55fb81bb9980 con 0x55fb817c9590 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x55fb81be2c40 con 0x55fb817ea270 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06ecdb4740 10 client.0 _ll_register_callbacks cb 0x7f06d4001220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06df7fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43308/0 (socket says 172.21.15.150:43308) 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06df7fe640 1 -- 172.21.15.150:0/3743835215 learned_addr learned my addr 172.21.15.150:0/3743835215 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06deffd640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c9590 0x55fb81bb6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.863+0000 7f06dffff640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817ea270 0x55fb81bd8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 -- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817ea270 msgr2=0x55fb81bd8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817ea270 0x55fb81bd8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 -- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c9590 msgr2=0x55fb81bb6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c9590 0x55fb81bb6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 -- 172.21.15.150:0/3743835215 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb81bd3960 con 0x55fb817c13b0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06df7fe640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f06d00b42c0 tx=0x7f06d0009210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fdc67acded5c46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 10 client.0 ms_handle_connect on v2:172.21.15.150:3300/0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d00a1060 con 0x55fb817c13b0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mon.0 v2:172.21.15.150:3300/0 2 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f06d007f340 con 0x55fb817c13b0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f06d8007e10 con 0x55fb817c13b0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f06d007fee0 con 0x55fb817c13b0 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 10 client.6036 fetch_fsmap learned FSMap version 37 2024-04-25T14:06:38.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=2+}) -- 0x55fb81bac2c0 con 0x55fb817c13b0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mon.0 v2:172.21.15.150:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d8007e10 con 0x55fb817c13b0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mon.0 v2:172.21.15.150:3300/0 5 ==== fsmap(e 37) ==== 3494+0+0 (secure 0 0 0) 0x7f06d007b850 con 0x55fb817c13b0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 10 client.6036 fetch_fsmap finished waiting for FSMap version 37 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 10 client.6036 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 20 client.6036 populate_metadata read hostname 'smithi150' 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb81bc6de0 0x55fb81bc71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 4 client.6036 mds_command: new command op to 5982 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 --> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] -- command(tid 0: {"prefix": "status"}) -- 0x55fb81af81c0 con 0x55fb81bc6de0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.864+0000 7f06deffd640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb81bc6de0 0x55fb81bc71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06be7fc640 10 client.6036 ms_handle_connect on v2:172.21.15.150:6834/685573768 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06deffd640 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb81bc6de0 0x55fb81bc71f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20724253d8fa56d server_cookie=a119d0e6c761dac in_seq=0 out_seq=0 2024-04-25T14:06:38.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06be7fc640 1 -- 172.21.15.150:0/3743835215 <== mds.0 v2:172.21.15.150:6834/685573768 1 ==== command_reply(tid 0: 0 ) ==== 8+0+320 (crc 0 0 0) 0x55fb81af81c0 con 0x55fb81bc6de0 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06be7fc640 10 client.6036 handle_command_reply: tid=0 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 client.6036 shutdown 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 msgr2=0x55fb81bb5e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f06d00b42c0 tx=0x7f06d0009210 comp rx=0 tx=0).stop 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 shutdown_connections 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:6834/685573768,v1:172.21.15.150:6836/685573768] conn(0x55fb81bc6de0 0x55fb81bc71f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x55fb817ea270 0x55fb81bd8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x55fb817c9590 0x55fb81bb6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06be7fc640 0 client.6036 ms_handle_reset on v2:172.21.15.150:6834/685573768 2024-04-25T14:06:38.872 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 --2- 172.21.15.150:0/3743835215 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x55fb817c13b0 0x55fb81bb5e30 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 >> 172.21.15.150:0/3743835215 conn(0x55fb817de3c0 msgr2=0x55fb81b8ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 shutdown_connections 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.865+0000 7f06ecdb4740 1 -- 172.21.15.150:0/3743835215 wait complete. 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06ecdb4740 20 client.6036 trim_cache size 0 max 16384 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stdout: "cluster_fsid": "77163e42-55ae-4cab-b506-bd8c531cc087", 2024-04-25T14:06:38.873 INFO:teuthology.orchestra.run.smithi150.stdout: "whoami": 0, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "id": 5982, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "want_state": "up:active", 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "state": "up:active", 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "fs_name": "cephfs", 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "rank_uptime": 121.603384674, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "mdsmap_epoch": 37, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch": 62, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "osdmap_epoch_barrier": 60, 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout: "uptime": 122.62715846 2024-04-25T14:06:38.874 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:06:38.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 msgr2=0x7f06cc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 0x7f06cc0769c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f06d4007fd0 tx=0x7f06d4009040 comp rx=0 tx=0).stop 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 msgr2=0x7f06e01f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f06d002a270 tx=0x7f06d00046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 shutdown_connections 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f06cc0744b0 0x7f06cc0769c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f06e01622d0 0x7f06e006dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.876 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f06e015fe40 0x7f06e00c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 --2- 172.21.15.150:0/986382437 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f06e00cff20 0x7f06e01f6270 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 >> 172.21.15.150:0/986382437 conn(0x7f06e00b76d0 msgr2=0x7f06e00ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 shutdown_connections 2024-04-25T14:06:38.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.866+0000 7f06e5f00640 1 -- 172.21.15.150:0/986382437 wait complete. 2024-04-25T14:06:38.878 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:38.986 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.986+0000 7f8f951e0640 1 Processor -- start 2024-04-25T14:06:38.987 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.986+0000 7f8f951e0640 1 -- start start 2024-04-25T14:06:38.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f951e0640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 0x7f8f901648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.988 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f951e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f951e0640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f90160810 con 0x7f8f900d1ae0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f951e0640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8f900d0300 con 0x7f8f900cebe0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f951e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f90164e30 con 0x7f8f900b45f0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8ed9d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 0x7f8f901648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:33374/0 (socket says 172.21.15.150:33374) 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 -- 172.21.15.150:0/2279750944 learned_addr learned my addr 172.21.15.150:0/2279750944 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8ed9d640 1 --2- 172.21.15.150:0/2279750944 >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 0x7f8f901648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3300/0 says I am v2:172.21.15.150:59862/0 (socket says 172.21.15.150:59862) 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8dd9b640 1 -- 172.21.15.150:0/2279750944 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404534958 0 0) 0x7f8f90160810 con 0x7f8f900d1ae0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8dd9b640 1 -- 172.21.15.150:0/2279750944 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f78003520 con 0x7f8f900d1ae0 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 -- 172.21.15.150:0/2279750944 >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 msgr2=0x7f8f901648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 --2- 172.21.15.150:0/2279750944 >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 0x7f8f901648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 -- 172.21.15.150:0/2279750944 >> v1:172.21.15.187:6789/0 conn(0x7f8f900d1ae0 legacy=0x7f8f900d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.988+0000 7f8f8e59c640 1 -- 172.21.15.150:0/2279750944 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f900b4f40 con 0x7f8f900b45f0 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.989+0000 7f8f8e59c640 1 --2- 172.21.15.150:0/2279750944 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009870 tx=0x7f8f8402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e3607bb01b6800 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.989+0000 7f8f8dd9b640 1 -- 172.21.15.150:0/2279750944 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f84037070 con 0x7f8f900b45f0 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.989+0000 7f8f8dd9b640 1 -- 172.21.15.150:0/2279750944 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f84032440 con 0x7f8f900b45f0 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.989+0000 7f8f8dd9b640 1 -- 172.21.15.150:0/2279750944 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f840327f0 con 0x7f8f900b45f0 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 -- 172.21.15.150:0/2279750944 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 msgr2=0x7f8f900b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2279750944 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009870 tx=0x7f8f8402a270 comp rx=0 tx=0).stop 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 -- 172.21.15.150:0/2279750944 shutdown_connections 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2279750944 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900b45f0 0x7f8f900b4a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2279750944 >> v2:172.21.15.187:3300/0 conn(0x7f8f900cebe0 0x7f8f901648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 -- 172.21.15.150:0/2279750944 >> 172.21.15.150:0/2279750944 conn(0x7f8f900b76d0 msgr2=0x7f8f900b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:38.991 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 -- 172.21.15.150:0/2279750944 shutdown_connections 2024-04-25T14:06:38.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.990+0000 7f8f951e0640 1 -- 172.21.15.150:0/2279750944 wait complete. 2024-04-25T14:06:38.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.992+0000 7f8f951e0640 1 Processor -- start 2024-04-25T14:06:38.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.992+0000 7f8f951e0640 1 -- start start 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f900cebe0 0x7f8f9015eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900d1ae0 0x7f8f90145c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8f900cfe10 con 0x7f8f900b45f0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8f900b4160 con 0x7f8f900cebe0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f951e0640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8f900cfb30 con 0x7f8f900d1ae0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f8ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.993+0000 7f8f8ed9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43310/0 (socket says 172.21.15.150:43310) 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 -- 172.21.15.150:0/2167880303 learned_addr learned my addr 172.21.15.150:0/2167880303 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8e59c640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f900cebe0 0x7f8f9015eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8f59e640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900d1ae0 0x7f8f90145c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 -- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900d1ae0 msgr2=0x7f8f90145c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.995 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900d1ae0 0x7f8f90145c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 -- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f900cebe0 msgr2=0x7f8f9015eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f900cebe0 0x7f8f9015eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 -- 172.21.15.150:0/2167880303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f90146980 con 0x7f8f900b45f0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f8ed9d640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009dc0 tx=0x7f8f8000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80018020 con 0x7f8f900b45f0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f80014480 con 0x7f8f900b45f0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f900c8590 con 0x7f8f900b45f0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80014850 con 0x7f8f900b45f0 2024-04-25T14:06:38.996 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.994+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f900c8b50 con 0x7f8f900b45f0 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.995+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f90050740 con 0x7f8f900b45f0 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.996+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8f8001b020 con 0x7f8f900b45f0 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.996+0000 7f8f777fe640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 0x7f8f7c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.997+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f8f80096240 con 0x7f8f900b45f0 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.997+0000 7f8f8e59c640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 0x7f8f7c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:38.997 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:38.997+0000 7f8f8e59c640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 0x7f8f7c0769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009870 tx=0x7f8f8402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:39.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.002+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f80022020 con 0x7f8f900b45f0 2024-04-25T14:06:39.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.213+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f900cf590 con 0x7f8f900b45f0 2024-04-25T14:06:39.213 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.213+0000 7f8f777fe640 1 -- 172.21.15.150:0/2167880303 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f8f80010040 con 0x7f8f900b45f0 2024-04-25T14:06:39.214 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:39.214 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:39.214 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 msgr2=0x7f8f7c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 0x7f8f7c0769c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8f84009870 tx=0x7f8f8402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 msgr2=0x7f8f900c9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8f80009dc0 tx=0x7f8f8000c490 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 shutdown_connections 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8f7c0744b0 0x7f8f7c0769c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8f900d1ae0 0x7f8f90145c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8f900cebe0 0x7f8f9015eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 --2- 172.21.15.150:0/2167880303 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8f900b45f0 0x7f8f900c9070 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:39.216 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 >> 172.21.15.150:0/2167880303 conn(0x7f8f900b76d0 msgr2=0x7f8f90071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:39.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 shutdown_connections 2024-04-25T14:06:39.217 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:39.215+0000 7f8f951e0640 1 -- 172.21.15.150:0/2167880303 wait complete. 2024-04-25T14:06:39.227 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:39.352 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":149,"msgr_send_messages":32,"msgr_recv_bytes":64092,"msgr_send_bytes":31674,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.027236511,"msgr_running_send_time":0.007914004,"msgr_running_recv_time":0.010949648,"msgr_running_fast_dispatch_time":0.007568484,"msgr_send_messages_queue_lat":{"avgcount":31,"sum":0.006602382,"avgtime":0.000212980},"msgr_handle_ack_lat":{"avgcount":141,"sum":0.000032983,"avgtime":0.000000233},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":60,"msgr_send_messages":55,"msgr_recv_bytes":127400,"msgr_send_bytes":27834,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.008955767,"msgr_running_send_time":0.002908181,"msgr_running_recv_time":0.005411050,"msgr_running_fast_dispatch_time":0.001043664,"msgr_send_messages_queue_lat":{"avgcount":55,"sum":0.005677310,"avgtime":0.000103223},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123424,"msgr_send_encrypted_bytes":24416},"AsyncMessenger::Worker-2":{"msgr_recv_messages":25,"msgr_send_messages":51,"msgr_recv_bytes":13846,"msgr_send_bytes":141390,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006114841,"msgr_running_send_time":0.002451544,"msgr_running_recv_time":0.002654911,"msgr_running_fast_dispatch_time":0.000585698,"msgr_send_messages_queue_lat":{"avgcount":50,"sum":0.004233101,"avgtime":0.000084662},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":131632},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":49,"sum":0.004083435,"avgtime":0.000083335}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":2,"reply":0,"reply_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":10,"inodes_top":7,"inodes_bottom":3,"inodes_pin_tail":0,"inodes_pinned":10,"inodes_expired":0,"inodes_with_caps":1,"caps":1,"subtrees":2,"traverse":1,"traverse_hit":1,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":0,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":0},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":2,"evex":0,"evtrm":0,"ev":5,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4196431,"rdpos":4195244,"jlat":{"avgcount":2,"sum":0.004119841,"avgtime":0.002059920},"replayed":3},"mds_mem":{"ino":13,"ino+":13,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":10,"dn+":10,"dn-":0,"cap":1,"cap+":1,"cap-":0,"rss":114452,"heap":248188},"mds_server":{"dispatch_client_request":1,"dispatch_server_request":0,"handle_client_request":2,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":0,"average_load":0,"avg_session_uptime":721,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237031,"buffer_anon_items":67,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":52528,"mds_co_items":189,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":47,"op_send_bytes":1559,"op_resend":0,"op_reply":47,"op_latency":{"avgcount":47,"sum":0.070998680,"avgtime":0.001510610},"op_inflight":0,"oplen_avg":{"avgcount":47,"sum":94},"op":47,"op_r":36,"op_w":11,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":2,"osdop_writefull":1,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":3,"omap_rd":48,"omap_del":0},"oft":{"omap_total_objs":0,"omap_total_kv_pairs":0,"omap_total_updates":0,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":229,"get_sum":180518,"get_or_fail_fail":0,"get_or_fail_success":229,"take":0,"take_sum":0,"put":229,"put_sum":180518,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":47,"take_sum":2491,"put":47,"put_sum":2491,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":47,"take_sum":47,"put":47,"put_sum":47,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":2,"get_sum":931,"get_or_fail_fail":0,"get_or_fail_success":2,"take":0,"take_sum":0,"put":2,"put_sum":931,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:39.354 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 141, "avgtime": 2.33e-07, "sum": 3.2983e-05 }, "msgr_recv_bytes": 64092, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 149, "msgr_running_fast_dispatch_time": 0.007568484, "msgr_running_recv_time": 0.010949648, "msgr_running_send_time": 0.007914004, "msgr_running_total_time": 0.027236511, "msgr_send_bytes": 31674, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 32, "msgr_send_messages_queue_lat": { "avgcount": 31, "avgtime": 0.00021298, "sum": 0.006602382 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127400, "msgr_recv_encrypted_bytes": 123424, "msgr_recv_messages": 60, "msgr_running_fast_dispatch_time": 0.001043664, "msgr_running_recv_time": 0.00541105, "msgr_running_send_time": 0.002908181, "msgr_running_total_time": 0.008955767, "msgr_send_bytes": 27834, "msgr_send_encrypted_bytes": 24416, "msgr_send_messages": 55, "msgr_send_messages_queue_lat": { "avgcount": 55, "avgtime": 0.000103223, "sum": 0.00567731 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 13846, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 25, "msgr_running_fast_dispatch_time": 0.000585698, "msgr_running_recv_time": 0.002654911, "msgr_running_send_time": 0.002451544, "msgr_running_total_time": 0.006114841, "msgr_send_bytes": 141390, "msgr_send_encrypted_bytes": 131632, "msgr_send_messages": 51, "msgr_send_messages_queue_lat": { "avgcount": 50, "avgtime": 8.4662e-05, "sum": 0.004233101 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 49, "avgtime": 8.3335e-05, "sum": 0.004083435 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 1, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 10, "inodes_bottom": 3, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 10, "inodes_top": 7, "inodes_with_caps": 1, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 0, "q": 0, "reply": 0, "reply_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "request": 2, "root_rbytes": 0, "root_rfiles": 0, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 1, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 1, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 5, "evadd": 2, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 2, "avgtime": 0.00205992, "sum": 0.004119841 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4196431 }, "mds_mem": { "cap": 1, "cap+": 1, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 10, "dn+": 10, "dn-": 0, "heap": 248188, "ino": 13, "ino+": 13, "ino-": 0, "rss": 114452 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 1, "dispatch_server_request": 0, "handle_client_request": 2, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 0, "avg_session_uptime": 721, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 0 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237031, "buffer_anon_items": 67, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 52528, "mds_co_items": 189, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 0, "omap_rd": 48, "omap_wr": 3, "op": 47, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 47, "avgtime": 0.00151061, "sum": 0.07099868 }, "op_pg": 0, "op_r": 36, "op_reply": 47, "op_resend": 0, "op_rmw": 0, "op_send": 47, "op_send_bytes": 1559, "op_w": 11, "oplen_avg": { "avgcount": 47, "sum": 94 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 2, "osdop_writefull": 1, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 0, "omap_total_objs": 0, "omap_total_removes": 0, "omap_total_updates": 0 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 229, "get_or_fail_fail": 0, "get_or_fail_success": 229, "get_started": 0, "get_sum": 180518, "max": 104857600, "put": 229, "put_sum": 180518, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 47, "put_sum": 2491, "take": 47, "take_sum": 2491, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 47, "put_sum": 47, "take": 47, "take_sum": 47, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 2, "get_or_fail_fail": 0, "get_or_fail_success": 2, "get_started": 0, "get_sum": 931, "max": 3758096384, "put": 2, "put_sum": 931, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:39.354 INFO:tasks.cephfs.test_openfiletable:omap_total_updates_0:0 2024-04-25T14:06:39.354 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:39.354 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:39.370 DEBUG:teuthology.orchestra.run.smithi150:> stdin-killer -- python3 -c ' 2024-04-25T14:06:39.370 DEBUG:teuthology.orchestra.run.smithi150:> import fcntl 2024-04-25T14:06:39.370 DEBUG:teuthology.orchestra.run.smithi150:> import os 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> import sys 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> import time 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> fcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK) 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> with open("/home/ubuntu/cephtest/mnt.0/omap_counter_test_file", '"'"'w'"'"') as f: 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> f.write("content") 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> f.flush() 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> while True: 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> print("open_background: keeping file open", file=sys.stderr) 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> try: 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> if os.read(0, 4096) == b"": 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> break 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> except BlockingIOError: 2024-04-25T14:06:39.371 DEBUG:teuthology.orchestra.run.smithi150:> pass 2024-04-25T14:06:39.372 DEBUG:teuthology.orchestra.run.smithi150:> time.sleep(2) 2024-04-25T14:06:39.372 DEBUG:teuthology.orchestra.run.smithi150:> ' 2024-04-25T14:06:39.412 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/omap_counter_test_file 2024-04-25T14:06:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:06:39.426 INFO:teuthology.orchestra.run.smithi150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0/omap_counter_test_file': No such file or directory 2024-04-25T14:06:39.474 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:06:39.477 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:06:39.480 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:06:40.427 DEBUG:teuthology.orchestra.run.smithi150:> stat --printf=%s /home/ubuntu/cephtest/mnt.0/omap_counter_test_file 2024-04-25T14:06:40.446 INFO:teuthology.orchestra.run.smithi150.stdout:7 2024-04-25T14:06:40.446 INFO:tasks.cephfs.mount:File omap_counter_test_file became visible with size 7 from 0 after 1s 2024-04-25T14:06:40.447 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:40.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.596+0000 7f94d292e640 1 Processor -- start 2024-04-25T14:06:40.597 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.596+0000 7f94d292e640 1 -- start start 2024-04-25T14:06:40.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d292e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94cc0cebe0 0x7f94cc1648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d292e640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f94cc0d1ae0 0x7f94cc0d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.600 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d292e640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94cc160810 con 0x7f94cc0b45f0 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d292e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f94cc0d0300 con 0x7f94cc0cebe0 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d292e640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f94cc164e30 con 0x7f94cc0d1ae0 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94cc0cebe0 0x7f94cc1648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cb7fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f94cc0b45f0 0x7f94cc0b4a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41770/0 (socket says 172.21.15.150:41770) 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f94cc0cebe0 0x7f94cc1648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43326/0 (socket says 172.21.15.150:43326) 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cb7fe640 1 -- 172.21.15.150:0/2091259126 learned_addr learned my addr 172.21.15.150:0/2091259126 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94d0ecb640 1 --2- 172.21.15.150:0/2091259126 >> v2:172.21.15.187:3300/0 conn(0x7f94cc0d1ae0 0x7f94cc0d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 -- 172.21.15.150:0/2091259126 >> v2:172.21.15.187:3300/0 conn(0x7f94cc0d1ae0 msgr2=0x7f94cc0d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 --2- 172.21.15.150:0/2091259126 >> v2:172.21.15.187:3300/0 conn(0x7f94cc0d1ae0 0x7f94cc0d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 -- 172.21.15.150:0/2091259126 >> v1:172.21.15.150:6789/0 conn(0x7f94cc0b45f0 legacy=0x7f94cc0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94cbfff640 1 -- 172.21.15.150:0/2091259126 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94cc0cf850 con 0x7f94cc0cebe0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.598+0000 7f94caffd640 1 -- 172.21.15.150:0/2091259126 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755972912 0 0) 0x7f94cc160810 con 0x7f94cc0b45f0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.599+0000 7f94cbfff640 1 --2- 172.21.15.150:0/2091259126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1648f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f94cc075420 tx=0x7f94bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3ce423265bf3af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.599+0000 7f94caffd640 1 -- 172.21.15.150:0/2091259126 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc037070 con 0x7f94cc0cebe0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.599+0000 7f94caffd640 1 -- 172.21.15.150:0/2091259126 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc032440 con 0x7f94cc0cebe0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.599+0000 7f94caffd640 1 -- 172.21.15.150:0/2091259126 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc032810 con 0x7f94cc0cebe0 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 -- 172.21.15.150:0/2091259126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0cebe0 msgr2=0x7f94cc1648f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 --2- 172.21.15.150:0/2091259126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1648f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f94cc075420 tx=0x7f94bc02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.602 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 -- 172.21.15.150:0/2091259126 shutdown_connections 2024-04-25T14:06:40.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 --2- 172.21.15.150:0/2091259126 >> v2:172.21.15.187:3300/0 conn(0x7f94cc0d1ae0 0x7f94cc0d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 --2- 172.21.15.150:0/2091259126 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1648f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 -- 172.21.15.150:0/2091259126 >> 172.21.15.150:0/2091259126 conn(0x7f94cc0b76d0 msgr2=0x7f94cc0b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:40.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 -- 172.21.15.150:0/2091259126 shutdown_connections 2024-04-25T14:06:40.603 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.600+0000 7f94d292e640 1 -- 172.21.15.150:0/2091259126 wait complete. 2024-04-25T14:06:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.602+0000 7f94d292e640 1 Processor -- start 2024-04-25T14:06:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.602+0000 7f94d292e640 1 -- start start 2024-04-25T14:06:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1ea560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 0x7f94cc1eaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc0cfe10 con 0x7f94cc0b45f0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f94cc0b4160 con 0x7f94cc0cebe0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d292e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f94cc0b3df0 con 0x7f94cc0d1ae0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43332/0 (socket says 172.21.15.150:43332) 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 -- 172.21.15.150:0/3017850909 learned_addr learned my addr 172.21.15.150:0/3017850909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d0ecb640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 0x7f94cc1eaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cb7fe640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 -- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 msgr2=0x7f94cc1eaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 0x7f94cc1eaaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 -- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 msgr2=0x7f94cc1ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cbfff640 1 -- 172.21.15.150:0/3017850909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94cc0c63e0 con 0x7f94cc0b45f0 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94d0ecb640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 0x7f94cc1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:40.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.603+0000 7f94cb7fe640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94cbfff640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94bc035040 tx=0x7f94bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc037040 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94bc042070 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94cc15bea0 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94bc006030 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.604+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94cc15c180 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.605+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94cc0c65b0 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.605+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f94bc0061d0 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.606+0000 7f94c8ff9640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 0x7f94b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.606+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f94bc0b5b10 con 0x7f94cc0b45f0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.606+0000 7f94cb7fe640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 0x7f94b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:40.608 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.606+0000 7f94cb7fe640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 0x7f94b80769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0713d0 tx=0x7f94c0007cc0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:40.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.611+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94bc080090 con 0x7f94cc0b45f0 2024-04-25T14:06:40.822 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.822+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f94cc050740 con 0x7f94cc0b45f0 2024-04-25T14:06:40.823 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.823+0000 7f94c8ff9640 1 -- 172.21.15.150:0/3017850909 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f94bc088080 con 0x7f94cc0b45f0 2024-04-25T14:06:40.823 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:40.823 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:40.824 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:40.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.824+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 msgr2=0x7f94b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.825 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.824+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 0x7f94b80769c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0713d0 tx=0x7f94c0007cc0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 msgr2=0x7f94cc1f65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f94bc035040 tx=0x7f94bc0046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 shutdown_connections 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f94b80744b0 0x7f94b80769c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f94cc0d1ae0 0x7f94cc1eaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f94cc0cebe0 0x7f94cc1ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 --2- 172.21.15.150:0/3017850909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f94cc0b45f0 0x7f94cc1f65b0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 >> 172.21.15.150:0/3017850909 conn(0x7f94cc0b76d0 msgr2=0x7f94cc06d2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 shutdown_connections 2024-04-25T14:06:40.826 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:40.825+0000 7f94d292e640 1 -- 172.21.15.150:0/3017850909 wait complete. 2024-04-25T14:06:40.836 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:40.960 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":156,"msgr_send_messages":37,"msgr_recv_bytes":66611,"msgr_send_bytes":34736,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.028018971,"msgr_running_send_time":0.008230284,"msgr_running_recv_time":0.011194720,"msgr_running_fast_dispatch_time":0.007769811,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.006808923,"avgtime":0.000189136},"msgr_handle_ack_lat":{"avgcount":145,"sum":0.000034222,"avgtime":0.000000236},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":61,"msgr_send_messages":56,"msgr_recv_bytes":127672,"msgr_send_bytes":28474,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009072145,"msgr_running_send_time":0.002976010,"msgr_running_recv_time":0.005429068,"msgr_running_fast_dispatch_time":0.001062477,"msgr_send_messages_queue_lat":{"avgcount":56,"sum":0.005738792,"avgtime":0.000102478},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123696,"msgr_send_encrypted_bytes":25056},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":53,"msgr_recv_bytes":14338,"msgr_send_bytes":143905,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006267262,"msgr_running_send_time":0.002488596,"msgr_running_recv_time":0.002702820,"msgr_running_fast_dispatch_time":0.000643544,"msgr_send_messages_queue_lat":{"avgcount":52,"sum":0.004311986,"avgtime":0.000082922},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":131632},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":54,"sum":0.004581028,"avgtime":0.000084833}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":3,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4198220,"rdpos":4195244,"jlat":{"avgcount":3,"sum":0.006110290,"avgtime":0.002036763},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":114836,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":721,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237031,"buffer_anon_items":67,"buffer_meta_bytes":0,"buffer_meta_items":0,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":52528,"mds_co_items":189,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":53,"op_send_bytes":3974,"op_resend":0,"op_reply":53,"op_latency":{"avgcount":53,"sum":0.081998476,"avgtime":0.001547141},"op_inflight":0,"oplen_avg":{"avgcount":53,"sum":107},"op":53,"op_r":36,"op_w":17,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":3,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":2,"omap_total_kv_pairs":2,"omap_total_updates":2,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":239,"get_sum":182849,"get_or_fail_fail":0,"get_or_fail_success":239,"take":0,"take_sum":0,"put":239,"put_sum":182849,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":4906,"put":53,"put_sum":4906,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":53,"put":53,"put_sum":53,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":3,"get_sum":2720,"get_or_fail_fail":0,"get_or_fail_success":3,"take":0,"take_sum":0,"put":3,"put_sum":2720,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:40.962 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 145, "avgtime": 2.36e-07, "sum": 3.4222e-05 }, "msgr_recv_bytes": 66611, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 156, "msgr_running_fast_dispatch_time": 0.007769811, "msgr_running_recv_time": 0.01119472, "msgr_running_send_time": 0.008230284, "msgr_running_total_time": 0.028018971, "msgr_send_bytes": 34736, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 37, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.000189136, "sum": 0.006808923 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127672, "msgr_recv_encrypted_bytes": 123696, "msgr_recv_messages": 61, "msgr_running_fast_dispatch_time": 0.001062477, "msgr_running_recv_time": 0.005429068, "msgr_running_send_time": 0.00297601, "msgr_running_total_time": 0.009072145, "msgr_send_bytes": 28474, "msgr_send_encrypted_bytes": 25056, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 56, "avgtime": 0.000102478, "sum": 0.005738792 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14338, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000643544, "msgr_running_recv_time": 0.00270282, "msgr_running_send_time": 0.002488596, "msgr_running_total_time": 0.006267262, "msgr_send_bytes": 143905, "msgr_send_encrypted_bytes": 131632, "msgr_send_messages": 53, "msgr_send_messages_queue_lat": { "avgcount": 52, "avgtime": 8.2922e-05, "sum": 0.004311986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 54, "avgtime": 8.4833e-05, "sum": 0.004581028 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 0, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 3, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 3, "avgtime": 0.002036763, "sum": 0.00611029 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4198220 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 114836 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 721, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237031, "buffer_anon_items": 67, "buffer_meta_bytes": 0, "buffer_meta_items": 0, "mds_co_bytes": 52528, "mds_co_items": 189, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 3, "omap_rd": 48, "omap_wr": 11, "op": 53, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 53, "avgtime": 0.001547141, "sum": 0.081998476 }, "op_pg": 0, "op_r": 36, "op_reply": 53, "op_resend": 0, "op_rmw": 0, "op_send": 53, "op_send_bytes": 3974, "op_w": 17, "oplen_avg": { "avgcount": 53, "sum": 107 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 3, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 2, "omap_total_objs": 2, "omap_total_removes": 0, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 239, "get_or_fail_fail": 0, "get_or_fail_success": 239, "get_started": 0, "get_sum": 182849, "max": 104857600, "put": 239, "put_sum": 182849, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 53, "put_sum": 4906, "take": 53, "take_sum": 4906, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 53, "put_sum": 53, "take": 53, "take_sum": 53, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 3, "get_or_fail_fail": 0, "get_or_fail_success": 3, "get_started": 0, "get_sum": 2720, "max": 3758096384, "put": 3, "put_sum": 2720, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:40.962 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-25T14:06:40.962 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:41.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.069+0000 7ffb08a92640 1 Processor -- start 2024-04-25T14:06:41.070 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.069+0000 7ffb08a92640 1 -- start start 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb08a92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffb04160880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb08a92640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb040d2720 con 0x7ffb0409b630 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb08a92640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb0409bc60 con 0x7ffb0415f950 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb08a92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb04158b20 con 0x7ffb04160470 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7ffb0409b630 0x7ffb040af280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41780/0 (socket says 172.21.15.150:41780) 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0259c640 1 -- 172.21.15.150:0/3834199482 learned_addr learned my addr 172.21.15.150:0/3834199482 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:41.075 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2351210182 0 0) 0x7ffb040d2720 con 0x7ffb0409b630 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffb04160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffaec003520 con 0x7ffb0409b630 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2103925651 0 0) 0x7ffaec003520 con 0x7ffb0409b630 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 msgr2=0x7ffb04160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffb04160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 >> v1:172.21.15.187:6789/0 conn(0x7ffb0415f950 legacy=0x7ffb0415fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb040d29c0 con 0x7ffb0409b630 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7ffaf4001a10 con 0x7ffb0409b630 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.071+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 >> v1:172.21.15.150:6789/0 conn(0x7ffb0409b630 legacy=0x7ffb040af280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.076 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec0039e0 0x7ffaec003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffaec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaec004d40 0x7ffaec005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04158b20 con 0x7ffaec004d40 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffaec00baf0 con 0x7ffaec0039e0 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffaec00bcf0 con 0x7ffb04160470 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaec004d40 0x7ffaec005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb0259c640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec0039e0 0x7ffaec003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb01d9b640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffaec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.077 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 -- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 msgr2=0x7ffaec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb04160470 0x7ffaec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 -- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec0039e0 msgr2=0x7ffaec003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec0039e0 0x7ffaec003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.072+0000 7ffb02d9d640 1 -- 172.21.15.150:0/3834199482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec003520 con 0x7ffaec004d40 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaec004d40 0x7ffaec005150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009a90 tx=0x7ffaec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2446fedad0f2721 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc036070 con 0x7ffaec004d40 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffafc031400 con 0x7ffaec004d40 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb0159a640 1 -- 172.21.15.150:0/3834199482 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc009690 con 0x7ffaec004d40 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb08a92640 1 -- 172.21.15.150:0/3834199482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaec004d40 msgr2=0x7ffaec005150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb08a92640 1 --2- 172.21.15.150:0/3834199482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffaec004d40 0x7ffaec005150 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009a90 tx=0x7ffaec005830 comp rx=0 tx=0).stop 2024-04-25T14:06:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.073+0000 7ffb02d9d640 1 -- 172.21.15.150:0/3834199482 reap_dead start 2024-04-25T14:06:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.074+0000 7ffb08a92640 1 -- 172.21.15.150:0/3834199482 shutdown_connections 2024-04-25T14:06:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.074+0000 7ffb08a92640 1 -- 172.21.15.150:0/3834199482 >> 172.21.15.150:0/3834199482 conn(0x7ffb04073dc0 msgr2=0x7ffb040b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.074+0000 7ffb08a92640 1 -- 172.21.15.150:0/3834199482 shutdown_connections 2024-04-25T14:06:41.079 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.074+0000 7ffb08a92640 1 -- 172.21.15.150:0/3834199482 wait complete. 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.075+0000 7ffb08a92640 1 Processor -- start 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.076+0000 7ffb08a92640 1 -- start start 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec004d40 0x7ffb0414fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb0406c5c0 0x7ffb040cc570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7ffb0409ba40 con 0x7ffb0406d4b0 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04160880 con 0x7ffaec004d40 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7ffb04159090 con 0x7ffb0406c5c0 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43354/0 (socket says 172.21.15.150:43354) 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 -- 172.21.15.150:0/4009253934 learned_addr learned my addr 172.21.15.150:0/4009253934 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb0259c640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec004d40 0x7ffb0414fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb02d9d640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb0406c5c0 0x7ffb040cc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 -- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb0406c5c0 msgr2=0x7ffb040cc570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb0406c5c0 0x7ffb040cc570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 -- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec004d40 msgr2=0x7ffb0414fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec004d40 0x7ffb0414fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 -- 172.21.15.150:0/4009253934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb0406df30 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb01d9b640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800f850 tx=0x7ffaf800fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800fe40 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf8018070 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.077+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb0406dc50 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.078+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf800d730 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.078+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb0415dd50 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.079+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb0406d9c0 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.079+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7ffaf800d8d0 con 0x7ffb0406d4b0 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.080+0000 7ffaf2ffd640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 0x7ffae40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.080+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7ffaf8095240 con 0x7ffb0406d4b0 2024-04-25T14:06:41.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.080+0000 7ffb0259c640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 0x7ffae40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.080+0000 7ffb0259c640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 0x7ffae40769c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb04160df0 tx=0x7ffaf4006040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.085 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.084+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffaf805f840 con 0x7ffb0406d4b0 2024-04-25T14:06:41.294 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.294+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb04050740 con 0x7ffb0406d4b0 2024-04-25T14:06:41.295 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.294+0000 7ffaf2ffd640 1 -- 172.21.15.150:0/4009253934 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7ffaf805f140 con 0x7ffb0406d4b0 2024-04-25T14:06:41.295 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:41.295 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:41.295 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.296+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 msgr2=0x7ffae40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.296+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 0x7ffae40769c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffb04160df0 tx=0x7ffaf4006040 comp rx=0 tx=0).stop 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.296+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 msgr2=0x7ffb040cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.296+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800f850 tx=0x7ffaf800fd10 comp rx=0 tx=0).stop 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 shutdown_connections 2024-04-25T14:06:41.297 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7ffae40744b0 0x7ffae40769c0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7ffb0406c5c0 0x7ffb040cc570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7ffb0406d4b0 0x7ffb040cc030 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 --2- 172.21.15.150:0/4009253934 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7ffaec004d40 0x7ffb0414fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 >> 172.21.15.150:0/4009253934 conn(0x7ffb04073dc0 msgr2=0x7ffb04070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 shutdown_connections 2024-04-25T14:06:41.298 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.297+0000 7ffb08a92640 1 -- 172.21.15.150:0/4009253934 wait complete. 2024-04-25T14:06:41.308 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:41.432 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":156,"msgr_send_messages":37,"msgr_recv_bytes":66611,"msgr_send_bytes":34736,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.028018971,"msgr_running_send_time":0.008230284,"msgr_running_recv_time":0.011194720,"msgr_running_fast_dispatch_time":0.007769811,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.006808923,"avgtime":0.000189136},"msgr_handle_ack_lat":{"avgcount":145,"sum":0.000034222,"avgtime":0.000000236},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":61,"msgr_send_messages":56,"msgr_recv_bytes":127672,"msgr_send_bytes":28474,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009072145,"msgr_running_send_time":0.002976010,"msgr_running_recv_time":0.005429068,"msgr_running_fast_dispatch_time":0.001062477,"msgr_send_messages_queue_lat":{"avgcount":56,"sum":0.005738792,"avgtime":0.000102478},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123696,"msgr_send_encrypted_bytes":25056},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":53,"msgr_recv_bytes":14338,"msgr_send_bytes":143905,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006267262,"msgr_running_send_time":0.002488596,"msgr_running_recv_time":0.002702820,"msgr_running_fast_dispatch_time":0.000643544,"msgr_send_messages_queue_lat":{"avgcount":52,"sum":0.004311986,"avgtime":0.000082922},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":131632},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":54,"sum":0.004581028,"avgtime":0.000084833}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":3,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4198220,"rdpos":4195244,"jlat":{"avgcount":3,"sum":0.006110290,"avgtime":0.002036763},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":114836,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":726,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237655,"buffer_anon_items":74,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":56056,"mds_co_items":248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":53,"op_send_bytes":3974,"op_resend":0,"op_reply":53,"op_latency":{"avgcount":53,"sum":0.081998476,"avgtime":0.001547141},"op_inflight":0,"oplen_avg":{"avgcount":53,"sum":107},"op":53,"op_r":36,"op_w":17,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":3,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":2,"omap_total_kv_pairs":2,"omap_total_updates":2,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":239,"get_sum":182849,"get_or_fail_fail":0,"get_or_fail_success":239,"take":0,"take_sum":0,"put":239,"put_sum":182849,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":4906,"put":53,"put_sum":4906,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":53,"put":53,"put_sum":53,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":3,"get_sum":2720,"get_or_fail_fail":0,"get_or_fail_success":3,"take":0,"take_sum":0,"put":3,"put_sum":2720,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:41.433 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 145, "avgtime": 2.36e-07, "sum": 3.4222e-05 }, "msgr_recv_bytes": 66611, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 156, "msgr_running_fast_dispatch_time": 0.007769811, "msgr_running_recv_time": 0.01119472, "msgr_running_send_time": 0.008230284, "msgr_running_total_time": 0.028018971, "msgr_send_bytes": 34736, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 37, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.000189136, "sum": 0.006808923 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127672, "msgr_recv_encrypted_bytes": 123696, "msgr_recv_messages": 61, "msgr_running_fast_dispatch_time": 0.001062477, "msgr_running_recv_time": 0.005429068, "msgr_running_send_time": 0.00297601, "msgr_running_total_time": 0.009072145, "msgr_send_bytes": 28474, "msgr_send_encrypted_bytes": 25056, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 56, "avgtime": 0.000102478, "sum": 0.005738792 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14338, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000643544, "msgr_running_recv_time": 0.00270282, "msgr_running_send_time": 0.002488596, "msgr_running_total_time": 0.006267262, "msgr_send_bytes": 143905, "msgr_send_encrypted_bytes": 131632, "msgr_send_messages": 53, "msgr_send_messages_queue_lat": { "avgcount": 52, "avgtime": 8.2922e-05, "sum": 0.004311986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 54, "avgtime": 8.4833e-05, "sum": 0.004581028 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 0, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 3, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 3, "avgtime": 0.002036763, "sum": 0.00611029 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4198220 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 114836 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 726, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237655, "buffer_anon_items": 74, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 56056, "mds_co_items": 248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 3, "omap_rd": 48, "omap_wr": 11, "op": 53, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 53, "avgtime": 0.001547141, "sum": 0.081998476 }, "op_pg": 0, "op_r": 36, "op_reply": 53, "op_resend": 0, "op_rmw": 0, "op_send": 53, "op_send_bytes": 3974, "op_w": 17, "oplen_avg": { "avgcount": 53, "sum": 107 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 3, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 2, "omap_total_objs": 2, "omap_total_removes": 0, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 239, "get_or_fail_fail": 0, "get_or_fail_success": 239, "get_started": 0, "get_sum": 182849, "max": 104857600, "put": 239, "put_sum": 182849, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 53, "put_sum": 4906, "take": 53, "take_sum": 4906, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 53, "put_sum": 53, "take": 53, "take_sum": 53, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 3, "get_or_fail_fail": 0, "get_or_fail_success": 3, "get_started": 0, "get_sum": 2720, "max": 3758096384, "put": 3, "put_sum": 2720, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:41.433 INFO:tasks.cephfs.test_openfiletable:omap_total_updates_1:2 2024-04-25T14:06:41.433 DEBUG:tasks.cephfs.mount:terminating RemoteProcess(client=, args=['stdin-killer', '--', 'python3', '-c', '\nimport fcntl\nimport os\nimport sys\nimport time\n\nfcntl.fcntl(sys.stdin, fcntl.F_SETFL, fcntl.fcntl(sys.stdin, fcntl.F_GETFL) | os.O_NONBLOCK)\n\nwith open("/home/ubuntu/cephtest/mnt.0/omap_counter_test_file", \'w\') as f:\n f.write("content")\n f.flush()\n while True:\n print("open_background: keeping file open", file=sys.stderr)\n try:\n if os.read(0, 4096) == b"":\n break\n except BlockingIOError:\n pass\n time.sleep(2)\n'], check_status=True, hostname='smithi150') 2024-04-25T14:06:41.435 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41 stdin-killer INFO: expiration expected; waiting 5 seconds for command to complete 2024-04-25T14:06:41.482 INFO:teuthology.orchestra.run.smithi150.stderr:open_background: keeping file open 2024-04-25T14:06:41.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T14:06:41.492 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:41.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.640+0000 7fc5766a8640 1 Processor -- start 2024-04-25T14:06:41.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.640+0000 7fc5766a8640 1 -- start start 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc5766a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc57006f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc5766a8640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5700cfd80 0x7fc5700d1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc5766a8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5700cf2b0 con 0x7fc5700d0190 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc5766a8640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc5700cecf0 con 0x7fc5700cfd80 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc5766a8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5700cefd0 con 0x7fc5700d34b0 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5700cfd80 0x7fc5700d1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc574c45640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc5700d0190 0x7fc5700b3ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41786/0 (socket says 172.21.15.150:41786) 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc5700cfd80 0x7fc5700d1c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43360/0 (socket says 172.21.15.150:43360) 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc574c45640 1 -- 172.21.15.150:0/1589074774 learned_addr learned my addr 172.21.15.150:0/1589074774 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56ffff640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc57006f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 -- 172.21.15.150:0/1589074774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 msgr2=0x7fc57006f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc57006f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.643 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 -- 172.21.15.150:0/1589074774 >> v1:172.21.15.150:6789/0 conn(0x7fc5700d0190 legacy=0x7fc5700b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56effd640 1 -- 172.21.15.150:0/1589074774 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4080876825 0 0) 0x7fc5700cf2b0 con 0x7fc5700d0190 2024-04-25T14:06:41.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.642+0000 7fc56f7fe640 1 -- 172.21.15.150:0/1589074774 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5700cd170 con 0x7fc5700cfd80 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.643+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700d1c00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc56402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e977a5763644f7b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.643+0000 7fc56effd640 1 -- 172.21.15.150:0/1589074774 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564037070 con 0x7fc5700cfd80 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.643+0000 7fc56effd640 1 -- 172.21.15.150:0/1589074774 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc564032440 con 0x7fc5700cfd80 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.643+0000 7fc56effd640 1 -- 172.21.15.150:0/1589074774 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc564032810 con 0x7fc5700cfd80 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 -- 172.21.15.150:0/1589074774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 msgr2=0x7fc5700d1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700d1c00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc56402a180 comp rx=0 tx=0).stop 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 -- 172.21.15.150:0/1589074774 shutdown_connections 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700d1c00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 --2- 172.21.15.150:0/1589074774 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc57006f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 -- 172.21.15.150:0/1589074774 >> 172.21.15.150:0/1589074774 conn(0x7fc57009b630 msgr2=0x7fc57009ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:41.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 -- 172.21.15.150:0/1589074774 shutdown_connections 2024-04-25T14:06:41.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.644+0000 7fc5766a8640 1 -- 172.21.15.150:0/1589074774 wait complete. 2024-04-25T14:06:41.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.646+0000 7fc5766a8640 1 Processor -- start 2024-04-25T14:06:41.646 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.646+0000 7fc5766a8640 1 -- start start 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 0x7fc5700c9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc570155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc5700b2130 con 0x7fc5700cfd80 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc5700b1fb0 con 0x7fc5700d0190 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc5766a8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc5700d2140 con 0x7fc5700d34b0 2024-04-25T14:06:41.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc56ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc56ffff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43372/0 (socket says 172.21.15.150:43372) 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.647+0000 7fc56ffff640 1 -- 172.21.15.150:0/3961386403 learned_addr learned my addr 172.21.15.150:0/3961386403 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc574c45640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc570155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 0x7fc5700c9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 -- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 msgr2=0x7fc570155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc570155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 -- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 msgr2=0x7fc5700c9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 0x7fc5700c9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 -- 172.21.15.150:0/3961386403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5700ca360 con 0x7fc5700cfd80 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 0x7fc5700c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56ffff640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc56000c7f0 tx=0x7fc56000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560015070 con 0x7fc5700cfd80 2024-04-25T14:06:41.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc560004220 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560007620 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5700ca640 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.648+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5700cac00 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.649+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5700cafe0 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.650+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc5600079d0 con 0x7fc5700cfd80 2024-04-25T14:06:41.651 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.650+0000 7fc56cff9640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 0x7fc55c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:41.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.650+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fc560095710 con 0x7fc5700cfd80 2024-04-25T14:06:41.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.650+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 0x7fc55c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:41.652 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.651+0000 7fc56f7fe640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 0x7fc55c0769c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc564034000 tx=0x7fc56402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:41.655 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.655+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5700cafe0 con 0x7fc5700cfd80 2024-04-25T14:06:41.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.866+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc5700c7050 con 0x7fc5700cfd80 2024-04-25T14:06:41.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.867+0000 7fc56cff9640 1 -- 172.21.15.150:0/3961386403 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7fc56005fda0 con 0x7fc5700cfd80 2024-04-25T14:06:41.867 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:41.867 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:41.867 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:41.869 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 msgr2=0x7fc55c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 0x7fc55c0769c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc564034000 tx=0x7fc56402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 msgr2=0x7fc5700cbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc56000c7f0 tx=0x7fc56000ccb0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 shutdown_connections 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc55c0744b0 0x7fc55c0769c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.870 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc5700d34b0 0x7fc570155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc5700d0190 0x7fc5700c9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.873 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 --2- 172.21.15.150:0/3961386403 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc5700cfd80 0x7fc5700cbd90 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 >> 172.21.15.150:0/3961386403 conn(0x7fc57009b630 msgr2=0x7fc570071a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 shutdown_connections 2024-04-25T14:06:41.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:41.869+0000 7fc5766a8640 1 -- 172.21.15.150:0/3961386403 wait complete. 2024-04-25T14:06:41.881 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:42.004 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":157,"msgr_send_messages":37,"msgr_recv_bytes":67029,"msgr_send_bytes":34736,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.028141416,"msgr_running_send_time":0.008267715,"msgr_running_recv_time":0.011228823,"msgr_running_fast_dispatch_time":0.007817415,"msgr_send_messages_queue_lat":{"avgcount":36,"sum":0.006808923,"avgtime":0.000189136},"msgr_handle_ack_lat":{"avgcount":146,"sum":0.000034511,"avgtime":0.000000236},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":61,"msgr_send_messages":56,"msgr_recv_bytes":127672,"msgr_send_bytes":28474,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009072145,"msgr_running_send_time":0.002976010,"msgr_running_recv_time":0.005429068,"msgr_running_fast_dispatch_time":0.001062477,"msgr_send_messages_queue_lat":{"avgcount":56,"sum":0.005738792,"avgtime":0.000102478},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123696,"msgr_send_encrypted_bytes":25056},"AsyncMessenger::Worker-2":{"msgr_recv_messages":27,"msgr_send_messages":53,"msgr_recv_bytes":14338,"msgr_send_bytes":143905,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006267262,"msgr_running_send_time":0.002488596,"msgr_running_recv_time":0.002702820,"msgr_running_fast_dispatch_time":0.000643544,"msgr_send_messages_queue_lat":{"avgcount":52,"sum":0.004311986,"avgtime":0.000082922},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":131632},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":54,"sum":0.004581028,"avgtime":0.000084833}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":0,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":1,"handle_client_caps_dirty":0,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":3,"evex":0,"evtrm":0,"ev":6,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4198220,"rdpos":4195244,"jlat":{"avgcount":3,"sum":0.006110290,"avgtime":0.002036763},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":114836,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":726,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237655,"buffer_anon_items":74,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":56056,"mds_co_items":248,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":53,"op_send_bytes":3974,"op_resend":0,"op_reply":53,"op_latency":{"avgcount":53,"sum":0.081998476,"avgtime":0.001547141},"op_inflight":0,"oplen_avg":{"avgcount":53,"sum":107},"op":53,"op_r":36,"op_w":17,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":3,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":11,"omap_rd":48,"omap_del":3},"oft":{"omap_total_objs":2,"omap_total_kv_pairs":2,"omap_total_updates":2,"omap_total_removes":0},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":240,"get_sum":183177,"get_or_fail_fail":0,"get_or_fail_success":240,"take":0,"take_sum":0,"put":240,"put_sum":183177,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":4906,"put":53,"put_sum":4906,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":53,"take_sum":53,"put":53,"put_sum":53,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":3,"get_sum":2720,"get_or_fail_fail":0,"get_or_fail_success":3,"take":0,"take_sum":0,"put":3,"put_sum":2720,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:42.005 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 146, "avgtime": 2.36e-07, "sum": 3.4511e-05 }, "msgr_recv_bytes": 67029, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 157, "msgr_running_fast_dispatch_time": 0.007817415, "msgr_running_recv_time": 0.011228823, "msgr_running_send_time": 0.008267715, "msgr_running_total_time": 0.028141416, "msgr_send_bytes": 34736, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 37, "msgr_send_messages_queue_lat": { "avgcount": 36, "avgtime": 0.000189136, "sum": 0.006808923 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127672, "msgr_recv_encrypted_bytes": 123696, "msgr_recv_messages": 61, "msgr_running_fast_dispatch_time": 0.001062477, "msgr_running_recv_time": 0.005429068, "msgr_running_send_time": 0.00297601, "msgr_running_total_time": 0.009072145, "msgr_send_bytes": 28474, "msgr_send_encrypted_bytes": 25056, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 56, "avgtime": 0.000102478, "sum": 0.005738792 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14338, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 27, "msgr_running_fast_dispatch_time": 0.000643544, "msgr_running_recv_time": 0.00270282, "msgr_running_send_time": 0.002488596, "msgr_running_total_time": 0.006267262, "msgr_send_bytes": 143905, "msgr_send_encrypted_bytes": 131632, "msgr_send_messages": 53, "msgr_send_messages_queue_lat": { "avgcount": 52, "avgtime": 8.2922e-05, "sum": 0.004311986 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 54, "avgtime": 8.4833e-05, "sum": 0.004581028 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 1, "handle_client_caps_dirty": 0, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 0, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 6, "evadd": 3, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 3, "avgtime": 0.002036763, "sum": 0.00611029 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4198220 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 114836 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 726, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237655, "buffer_anon_items": 74, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 56056, "mds_co_items": 248, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 3, "omap_rd": 48, "omap_wr": 11, "op": 53, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 53, "avgtime": 0.001547141, "sum": 0.081998476 }, "op_pg": 0, "op_r": 36, "op_reply": 53, "op_resend": 0, "op_rmw": 0, "op_send": 53, "op_send_bytes": 3974, "op_w": 17, "oplen_avg": { "avgcount": 53, "sum": 107 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 3, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 2, "omap_total_objs": 2, "omap_total_removes": 0, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 240, "get_or_fail_fail": 0, "get_or_fail_success": 240, "get_started": 0, "get_sum": 183177, "max": 104857600, "put": 240, "put_sum": 183177, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 53, "put_sum": 4906, "take": 53, "take_sum": 4906, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 53, "put_sum": 53, "take": 53, "take_sum": 53, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 3, "get_or_fail_fail": 0, "get_or_fail_success": 3, "get_started": 0, "get_sum": 2720, "max": 3758096384, "put": 3, "put_sum": 2720, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:42.006 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=120 retry_count=0)... 2024-04-25T14:06:47.007 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:47.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.128+0000 7f5af8a9f640 1 Processor -- start 2024-04-25T14:06:47.129 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.129+0000 7f5af8a9f640 1 -- start start 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af8a9f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5af41602a0 0x7f5af416b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af8a9f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af40d5380 con 0x7f5af40d44a0 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af8a9f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af40d5610 con 0x7f5af40b9370 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af8a9f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f5af406c760 con 0x7f5af41602a0 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af259c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f5af40d44a0 0x7f5af406d240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:41796/0 (socket says 172.21.15.150:41796) 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af259c640 1 -- 172.21.15.150:0/2507049462 learned_addr learned my addr 172.21.15.150:0/2507049462 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115569262 0 0) 0x7f5af40d5380 con 0x7f5af40d44a0 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.130+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5adc003520 con 0x7f5af40d44a0 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af2d9d640 1 --2- 172.21.15.150:0/2507049462 >> v2:172.21.15.187:3300/0 conn(0x7f5af41602a0 0x7f5af416b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4008384968 0 0) 0x7f5adc003520 con 0x7f5af40d44a0 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 >> v2:172.21.15.187:3300/0 conn(0x7f5af41602a0 msgr2=0x7f5af416b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 --2- 172.21.15.150:0/2507049462 >> v2:172.21.15.187:3300/0 conn(0x7f5af41602a0 0x7f5af416b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 >> v1:172.21.15.187:6789/0 conn(0x7f5af40b9370 legacy=0x7f5af406d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af406ca40 con 0x7f5af40d44a0 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f5ae40019f0 con 0x7f5af40d44a0 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af2d9d640 1 --2- 172.21.15.150:0/2507049462 >> v2:172.21.15.187:3300/0 conn(0x7f5af41602a0 0x7f5af416b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 >> v1:172.21.15.150:6789/0 conn(0x7f5af40d44a0 legacy=0x7f5af406d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5adc003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5adc004800 0x7f5adc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5adc005300 0x7f5adc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5af406c760 con 0x7f5adc0039b0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5adc00c2a0 con 0x7f5adc005300 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5adc00c4a0 con 0x7f5adc004800 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5adc003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af1d9b640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5adc004800 0x7f5adc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af2d9d640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5adc005300 0x7f5adc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 -- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5adc004800 msgr2=0x7f5adc004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5adc004800 0x7f5adc004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 -- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5adc005300 msgr2=0x7f5adc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5adc005300 0x7f5adc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af259c640 1 -- 172.21.15.150:0/2507049462 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5adc003520 con 0x7f5adc0039b0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af2d9d640 1 -- 172.21.15.150:0/2507049462 reap_dead start 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.131+0000 7f5af1d9b640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5adc004800 0x7f5adc004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.132+0000 7f5af259c640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5adc003e20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400a860 tx=0x7f5adc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5910e2f6e307625b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.136 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.132+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400a480 con 0x7f5adc0039b0 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.132+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae402f070 con 0x7f5adc0039b0 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.132+0000 7f5af159a640 1 -- 172.21.15.150:0/2507049462 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae402bd20 con 0x7f5adc0039b0 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 -- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 msgr2=0x7f5adc003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5adc003e20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400a860 tx=0x7f5adc005830 comp rx=0 tx=0).stop 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 -- 172.21.15.150:0/2507049462 shutdown_connections 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/2507049462 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5adc003e20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 -- 172.21.15.150:0/2507049462 >> 172.21.15.150:0/2507049462 conn(0x7f5af40b78e0 msgr2=0x7f5af40b7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 -- 172.21.15.150:0/2507049462 shutdown_connections 2024-04-25T14:06:47.137 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.133+0000 7f5af8a9f640 1 -- 172.21.15.150:0/2507049462 wait complete. 2024-04-25T14:06:47.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.135+0000 7f5af8a9f640 1 Processor -- start 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.135+0000 7f5af8a9f640 1 -- start start 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5af40cb560 0x7f5af41f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5af4162890 0x7f5af4162ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f5af41606b0 con 0x7f5adc0039b0 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f5af40b9780 con 0x7f5af4162890 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af8a9f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f5af406d240 con 0x7f5af40cb560 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af259c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:43388/0 (socket says 172.21.15.150:43388) 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af259c640 1 -- 172.21.15.150:0/3110742021 learned_addr learned my addr 172.21.15.150:0/3110742021 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.136+0000 7f5af1d9b640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5af40cb560 0x7f5af41f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af2d9d640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5af4162890 0x7f5af4162ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 -- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5af40cb560 msgr2=0x7f5af41f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5af40cb560 0x7f5af41f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 -- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5af4162890 msgr2=0x7f5af4162ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5af4162890 0x7f5af4162ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 -- 172.21.15.150:0/3110742021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af41fae70 con 0x7f5adc0039b0 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af259c640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400a980 tx=0x7f5ae402d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4045070 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae4040680 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af41631e0 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400a480 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.137+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af40cbdc0 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.138+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5af4050740 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.139+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f5ae4048070 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.139+0000 7f5ae2ffd640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 0x7f5ad40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.139+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f5ae40b5850 con 0x7f5adc0039b0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.139+0000 7f5af1d9b640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 0x7f5ad40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.140+0000 7f5af1d9b640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 0x7f5ad40769c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5af4152d90 tx=0x7f5ae8009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.144 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.144+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5af41631e0 con 0x7f5adc0039b0 2024-04-25T14:06:47.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.354+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5af40d57e0 con 0x7f5adc0039b0 2024-04-25T14:06:47.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.354+0000 7f5ae2ffd640 1 -- 172.21.15.150:0/3110742021 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f5af40d57e0 con 0x7f5adc0039b0 2024-04-25T14:06:47.354 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:47.355 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:47.355 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 msgr2=0x7f5ad40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 0x7f5ad40769c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5af4152d90 tx=0x7f5ae8009040 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 msgr2=0x7f5af41fa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f5ae400a980 tx=0x7f5ae402d070 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 shutdown_connections 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f5ad40744b0 0x7f5ad40769c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f5af4162890 0x7f5af4162ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5af40cb560 0x7f5af41f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 --2- 172.21.15.150:0/3110742021 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f5adc0039b0 0x7f5af41fa930 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 >> 172.21.15.150:0/3110742021 conn(0x7f5af40b78e0 msgr2=0x7f5af40d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:47.357 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 shutdown_connections 2024-04-25T14:06:47.358 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.356+0000 7f5af8a9f640 1 -- 172.21.15.150:0/3110742021 wait complete. 2024-04-25T14:06:47.368 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:47.491 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":167,"msgr_send_messages":41,"msgr_recv_bytes":70697,"msgr_send_bytes":36340,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029497956,"msgr_running_send_time":0.008797377,"msgr_running_recv_time":0.011626490,"msgr_running_fast_dispatch_time":0.008219344,"msgr_send_messages_queue_lat":{"avgcount":40,"sum":0.006934539,"avgtime":0.000173363},"msgr_handle_ack_lat":{"avgcount":154,"sum":0.000036590,"avgtime":0.000000237},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":62,"msgr_send_messages":57,"msgr_recv_bytes":127944,"msgr_send_bytes":29114,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009171122,"msgr_running_send_time":0.003019201,"msgr_running_recv_time":0.005453740,"msgr_running_fast_dispatch_time":0.001085215,"msgr_send_messages_queue_lat":{"avgcount":57,"sum":0.005791126,"avgtime":0.000101598},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123968,"msgr_send_encrypted_bytes":25696},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":56,"msgr_recv_bytes":14584,"msgr_send_bytes":149579,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006522715,"msgr_running_send_time":0.002662324,"msgr_running_recv_time":0.002725140,"msgr_running_fast_dispatch_time":0.000675086,"msgr_send_messages_queue_lat":{"avgcount":55,"sum":0.004440028,"avgtime":0.000080727},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":135248},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":58,"sum":0.004854181,"avgtime":0.000083692}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":7,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2,"handle_client_caps_dirty":1,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4,"evex":0,"evtrm":0,"ev":7,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4199960,"rdpos":4195244,"jlat":{"avgcount":4,"sum":0.008043456,"avgtime":0.002010864},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":115220,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":731,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237655,"buffer_anon_items":74,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":56032,"mds_co_items":252,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":57,"op_send_bytes":5926,"op_resend":0,"op_reply":57,"op_latency":{"avgcount":57,"sum":0.089998328,"avgtime":0.001578918},"op_inflight":0,"oplen_avg":{"avgcount":57,"sum":113},"op":57,"op_r":36,"op_w":21,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":4,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":1,"omap_total_updates":2,"omap_total_removes":1},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":252,"get_sum":186231,"get_or_fail_fail":0,"get_or_fail_success":252,"take":0,"take_sum":0,"put":252,"put_sum":186231,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":6858,"put":57,"put_sum":6858,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":57,"put":57,"put_sum":57,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":4,"get_sum":4460,"get_or_fail_fail":0,"get_or_fail_success":4,"take":0,"take_sum":0,"put":4,"put_sum":4460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:47.493 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 154, "avgtime": 2.37e-07, "sum": 3.659e-05 }, "msgr_recv_bytes": 70697, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 167, "msgr_running_fast_dispatch_time": 0.008219344, "msgr_running_recv_time": 0.01162649, "msgr_running_send_time": 0.008797377, "msgr_running_total_time": 0.029497956, "msgr_send_bytes": 36340, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 41, "msgr_send_messages_queue_lat": { "avgcount": 40, "avgtime": 0.000173363, "sum": 0.006934539 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127944, "msgr_recv_encrypted_bytes": 123968, "msgr_recv_messages": 62, "msgr_running_fast_dispatch_time": 0.001085215, "msgr_running_recv_time": 0.00545374, "msgr_running_send_time": 0.003019201, "msgr_running_total_time": 0.009171122, "msgr_send_bytes": 29114, "msgr_send_encrypted_bytes": 25696, "msgr_send_messages": 57, "msgr_send_messages_queue_lat": { "avgcount": 57, "avgtime": 0.000101598, "sum": 0.005791126 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14584, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.000675086, "msgr_running_recv_time": 0.00272514, "msgr_running_send_time": 0.002662324, "msgr_running_total_time": 0.006522715, "msgr_send_bytes": 149579, "msgr_send_encrypted_bytes": 135248, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 55, "avgtime": 8.0727e-05, "sum": 0.004440028 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 58, "avgtime": 8.3692e-05, "sum": 0.004854181 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2, "handle_client_caps_dirty": 1, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 7, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7, "evadd": 4, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 4, "avgtime": 0.002010864, "sum": 0.008043456 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4199960 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 115220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 731, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237655, "buffer_anon_items": 74, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 56032, "mds_co_items": 252, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 4, "omap_rd": 48, "omap_wr": 15, "op": 57, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 57, "avgtime": 0.001578918, "sum": 0.089998328 }, "op_pg": 0, "op_r": 36, "op_reply": 57, "op_resend": 0, "op_rmw": 0, "op_send": 57, "op_send_bytes": 5926, "op_w": 21, "oplen_avg": { "avgcount": 57, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 4, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 1, "omap_total_objs": 1, "omap_total_removes": 1, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 252, "get_or_fail_fail": 0, "get_or_fail_success": 252, "get_started": 0, "get_sum": 186231, "max": 104857600, "put": 252, "put_sum": 186231, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 57, "put_sum": 6858, "take": 57, "take_sum": 6858, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 57, "put_sum": 57, "take": 57, "take_sum": 57, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 4, "get_or_fail_fail": 0, "get_or_fail_success": 4, "get_started": 0, "get_sum": 4460, "max": 3758096384, "put": 4, "put_sum": 4460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:47.493 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-04-25T14:06:47.493 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:47.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.601+0000 7f27c0e0c640 1 Processor -- start 2024-04-25T14:06:47.601 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.601+0000 7f27c0e0c640 1 -- start start 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27bc09b630 0x7f27bc0af280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 0x7f27bc15e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 0x7f27bc15f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f27bc158b20 con 0x7f27bc09b630 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f27bc09ba40 con 0x7f27bc15e5e0 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27c0e0c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27bc0d3010 con 0x7f27bc15ef30 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27bc09b630 0x7f27bc0af280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.604 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f27bc09b630 0x7f27bc0af280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47462/0 (socket says 172.21.15.150:47462) 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 -- 172.21.15.150:0/2146921959 learned_addr learned my addr 172.21.15.150:0/2146921959 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27bad9d640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 0x7f27bc15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 -- 172.21.15.150:0/2146921959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 msgr2=0x7f27bc15f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 0x7f27bc15f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 -- 172.21.15.150:0/2146921959 >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 msgr2=0x7f27bc15e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27b9d9b640 1 --2- 172.21.15.150:0/2146921959 >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 0x7f27bc15e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 --2- 172.21.15.150:0/2146921959 >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 0x7f27bc15e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27ba59c640 1 -- 172.21.15.150:0/2146921959 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27bc0af7c0 con 0x7f27bc09b630 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27bad9d640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 0x7f27bc15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.603+0000 7f27b9d9b640 1 --2- 172.21.15.150:0/2146921959 >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 0x7f27bc15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.604+0000 7f27ba59c640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc0af280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06f420 tx=0x7f27ac02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbd78d7639df32c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.604+0000 7f27b959a640 1 -- 172.21.15.150:0/2146921959 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac036070 con 0x7f27bc09b630 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.604+0000 7f27b959a640 1 -- 172.21.15.150:0/2146921959 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ac031600 con 0x7f27bc09b630 2024-04-25T14:06:47.605 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.604+0000 7f27b959a640 1 -- 172.21.15.150:0/2146921959 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac0319d0 con 0x7f27bc09b630 2024-04-25T14:06:47.606 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 -- 172.21.15.150:0/2146921959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 msgr2=0x7f27bc0af280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc0af280 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f27bc06f420 tx=0x7f27ac02a6c0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 -- 172.21.15.150:0/2146921959 shutdown_connections 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15ef30 0x7f27bc15f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/2146921959 >> v2:172.21.15.187:3300/0 conn(0x7f27bc15e5e0 0x7f27bc15e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/2146921959 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc0af280 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 -- 172.21.15.150:0/2146921959 >> 172.21.15.150:0/2146921959 conn(0x7f27bc073dc0 msgr2=0x7f27bc0b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 -- 172.21.15.150:0/2146921959 shutdown_connections 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.605+0000 7f27c0e0c640 1 -- 172.21.15.150:0/2146921959 wait complete. 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.607+0000 7f27c0e0c640 1 Processor -- start 2024-04-25T14:06:47.607 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.607+0000 7f27c0e0c640 1 -- start start 2024-04-25T14:06:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15e5e0 0x7f27bc14f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc15ef30 0x7f27bc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f27bc0d29c0 con 0x7f27bc09b630 2024-04-25T14:06:47.609 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f27bc0d2720 con 0x7f27bc15ef30 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27c0e0c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f27bc0d2ca0 con 0x7f27bc15e5e0 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47474/0 (socket says 172.21.15.150:47474) 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 -- 172.21.15.150:0/100021784 learned_addr learned my addr 172.21.15.150:0/100021784 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27b9d9b640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15e5e0 0x7f27bc14f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27bad9d640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc15ef30 0x7f27bc0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 -- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15e5e0 msgr2=0x7f27bc14f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15e5e0 0x7f27bc14f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 -- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc15ef30 msgr2=0x7f27bc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc15ef30 0x7f27bc0ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.608+0000 7f27ba59c640 1 -- 172.21.15.150:0/100021784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27bc14ff20 con 0x7f27bc09b630 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27ba59c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f27ac034040 tx=0x7f27ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.610 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac03f020 con 0x7f27bc09b630 2024-04-25T14:06:47.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27ac037040 con 0x7f27bc09b630 2024-04-25T14:06:47.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27bc0d0c80 con 0x7f27bc09b630 2024-04-25T14:06:47.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27ac036070 con 0x7f27bc09b630 2024-04-25T14:06:47.611 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.609+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27bc0d1240 con 0x7f27bc09b630 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.610+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27bc050740 con 0x7f27bc09b630 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.611+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f27ac042020 con 0x7f27bc09b630 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.611+0000 7f27aaffd640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 0x7f279c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.611+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f27ac0b5850 con 0x7f27bc09b630 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.611+0000 7f27b9d9b640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 0x7f279c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:47.612 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.612+0000 7f27b9d9b640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 0x7f279c0769c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f27bc15e4b0 tx=0x7f27b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:47.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.616+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27ac049030 con 0x7f27bc09b630 2024-04-25T14:06:47.830 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.830+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f27bc06fe80 con 0x7f27bc09b630 2024-04-25T14:06:47.831 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.831+0000 7f27aaffd640 1 -- 172.21.15.150:0/100021784 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f27bc14ff20 con 0x7f27bc09b630 2024-04-25T14:06:47.831 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:47.831 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:47.831 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 msgr2=0x7f279c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 0x7f279c0769c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f27bc15e4b0 tx=0x7f27b0008040 comp rx=0 tx=0).stop 2024-04-25T14:06:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 msgr2=0x7f27bc150120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f27ac034040 tx=0x7f27ac004a40 comp rx=0 tx=0).stop 2024-04-25T14:06:47.833 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 shutdown_connections 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f279c0744b0 0x7f279c0769c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f27bc15ef30 0x7f27bc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f27bc15e5e0 0x7f27bc14f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 --2- 172.21.15.150:0/100021784 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f27bc09b630 0x7f27bc150120 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 >> 172.21.15.150:0/100021784 conn(0x7f27bc073dc0 msgr2=0x7f27bc15cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 shutdown_connections 2024-04-25T14:06:47.834 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:47.833+0000 7f27c0e0c640 1 -- 172.21.15.150:0/100021784 wait complete. 2024-04-25T14:06:47.844 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:47.969 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":167,"msgr_send_messages":41,"msgr_recv_bytes":70697,"msgr_send_bytes":36340,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029497956,"msgr_running_send_time":0.008797377,"msgr_running_recv_time":0.011626490,"msgr_running_fast_dispatch_time":0.008219344,"msgr_send_messages_queue_lat":{"avgcount":40,"sum":0.006934539,"avgtime":0.000173363},"msgr_handle_ack_lat":{"avgcount":154,"sum":0.000036590,"avgtime":0.000000237},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":62,"msgr_send_messages":57,"msgr_recv_bytes":127944,"msgr_send_bytes":29114,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009171122,"msgr_running_send_time":0.003019201,"msgr_running_recv_time":0.005453740,"msgr_running_fast_dispatch_time":0.001085215,"msgr_send_messages_queue_lat":{"avgcount":57,"sum":0.005791126,"avgtime":0.000101598},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":123968,"msgr_send_encrypted_bytes":25696},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":56,"msgr_recv_bytes":14584,"msgr_send_bytes":149579,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006522715,"msgr_running_send_time":0.002662324,"msgr_running_recv_time":0.002725140,"msgr_running_fast_dispatch_time":0.000675086,"msgr_send_messages_queue_lat":{"avgcount":55,"sum":0.004440028,"avgtime":0.000080727},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":135248},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":58,"sum":0.004854181,"avgtime":0.000083692}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":7,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2,"handle_client_caps_dirty":1,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4,"evex":0,"evtrm":0,"ev":7,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4199960,"rdpos":4195244,"jlat":{"avgcount":4,"sum":0.008043456,"avgtime":0.002010864},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":115220,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":731,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237655,"buffer_anon_items":74,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":56032,"mds_co_items":252,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":57,"op_send_bytes":5926,"op_resend":0,"op_reply":57,"op_latency":{"avgcount":57,"sum":0.089998328,"avgtime":0.001578918},"op_inflight":0,"oplen_avg":{"avgcount":57,"sum":113},"op":57,"op_r":36,"op_w":21,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":4,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":1,"omap_total_updates":2,"omap_total_removes":1},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":252,"get_sum":186231,"get_or_fail_fail":0,"get_or_fail_success":252,"take":0,"take_sum":0,"put":252,"put_sum":186231,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":6858,"put":57,"put_sum":6858,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":57,"put":57,"put_sum":57,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":4,"get_sum":4460,"get_or_fail_fail":0,"get_or_fail_success":4,"take":0,"take_sum":0,"put":4,"put_sum":4460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:47.970 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 154, "avgtime": 2.37e-07, "sum": 3.659e-05 }, "msgr_recv_bytes": 70697, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 167, "msgr_running_fast_dispatch_time": 0.008219344, "msgr_running_recv_time": 0.01162649, "msgr_running_send_time": 0.008797377, "msgr_running_total_time": 0.029497956, "msgr_send_bytes": 36340, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 41, "msgr_send_messages_queue_lat": { "avgcount": 40, "avgtime": 0.000173363, "sum": 0.006934539 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 127944, "msgr_recv_encrypted_bytes": 123968, "msgr_recv_messages": 62, "msgr_running_fast_dispatch_time": 0.001085215, "msgr_running_recv_time": 0.00545374, "msgr_running_send_time": 0.003019201, "msgr_running_total_time": 0.009171122, "msgr_send_bytes": 29114, "msgr_send_encrypted_bytes": 25696, "msgr_send_messages": 57, "msgr_send_messages_queue_lat": { "avgcount": 57, "avgtime": 0.000101598, "sum": 0.005791126 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14584, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.000675086, "msgr_running_recv_time": 0.00272514, "msgr_running_send_time": 0.002662324, "msgr_running_total_time": 0.006522715, "msgr_send_bytes": 149579, "msgr_send_encrypted_bytes": 135248, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 55, "avgtime": 8.0727e-05, "sum": 0.004440028 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 58, "avgtime": 8.3692e-05, "sum": 0.004854181 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2, "handle_client_caps_dirty": 1, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 7, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7, "evadd": 4, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 4, "avgtime": 0.002010864, "sum": 0.008043456 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4199960 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 115220 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 731, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237655, "buffer_anon_items": 74, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 56032, "mds_co_items": 252, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 4, "omap_rd": 48, "omap_wr": 15, "op": 57, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 57, "avgtime": 0.001578918, "sum": 0.089998328 }, "op_pg": 0, "op_r": 36, "op_reply": 57, "op_resend": 0, "op_rmw": 0, "op_send": 57, "op_send_bytes": 5926, "op_w": 21, "oplen_avg": { "avgcount": 57, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 4, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 1, "omap_total_objs": 1, "omap_total_removes": 1, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 252, "get_or_fail_fail": 0, "get_or_fail_success": 252, "get_started": 0, "get_sum": 186231, "max": 104857600, "put": 252, "put_sum": 186231, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 57, "put_sum": 6858, "take": 57, "take_sum": 6858, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 57, "put_sum": 57, "take": 57, "take_sum": 57, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 4, "get_or_fail_fail": 0, "get_or_fail_success": 4, "get_started": 0, "get_sum": 4460, "max": 3758096384, "put": 4, "put_sum": 4460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:47.970 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-04-25T14:06:47.970 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:48.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.078+0000 7feccf71f640 1 Processor -- start 2024-04-25T14:06:48.078 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.078+0000 7feccf71f640 1 -- start start 2024-04-25T14:06:48.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccf71f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fecc80d44a0 0x7fecc80b9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccf71f640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecc80d5380 con 0x7fecc806d950 2024-04-25T14:06:48.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccf71f640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecc80d5610 con 0x7fecc81602a0 2024-04-25T14:06:48.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccf71f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fecc80b98b0 con 0x7fecc80d44a0 2024-04-25T14:06:48.080 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccd4bb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fecc80d44a0 0x7fecc80b9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccdcbc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fecc806d950 0x7fecc80d0b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37424/0 (socket says 172.21.15.150:37424) 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccd4bb640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fecc80d44a0 0x7fecc80b9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47476/0 (socket says 172.21.15.150:47476) 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccdcbc640 1 -- 172.21.15.150:0/2600139519 learned_addr learned my addr 172.21.15.150:0/2600139519 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccd4bb640 1 -- 172.21.15.150:0/2600139519 >> v1:172.21.15.187:6789/0 conn(0x7fecc81602a0 legacy=0x7fecc806d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccd4bb640 1 -- 172.21.15.150:0/2600139519 >> v1:172.21.15.150:6789/0 conn(0x7fecc806d950 legacy=0x7fecc80d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7fecbffff640 1 -- 172.21.15.150:0/2600139519 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859355503 0 0) 0x7fecc80d5380 con 0x7fecc806d950 2024-04-25T14:06:48.081 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.080+0000 7feccd4bb640 1 -- 172.21.15.150:0/2600139519 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc806c760 con 0x7fecc80d44a0 2024-04-25T14:06:48.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.081+0000 7feccd4bb640 1 --2- 172.21.15.150:0/2600139519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc80d44a0 0x7fecc80b9370 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fecc40097a0 tx=0x7fecc402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7a7df3a9df130e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.081+0000 7fecbffff640 1 -- 172.21.15.150:0/2600139519 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4037070 con 0x7fecc80d44a0 2024-04-25T14:06:48.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.081+0000 7fecbffff640 1 -- 172.21.15.150:0/2600139519 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc4032440 con 0x7fecc80d44a0 2024-04-25T14:06:48.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.081+0000 7fecbffff640 1 -- 172.21.15.150:0/2600139519 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4032810 con 0x7fecc80d44a0 2024-04-25T14:06:48.082 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 -- 172.21.15.150:0/2600139519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc80d44a0 msgr2=0x7fecc80b9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 --2- 172.21.15.150:0/2600139519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc80d44a0 0x7fecc80b9370 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fecc40097a0 tx=0x7fecc402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 -- 172.21.15.150:0/2600139519 shutdown_connections 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 --2- 172.21.15.150:0/2600139519 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc80d44a0 0x7fecc80b9370 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 -- 172.21.15.150:0/2600139519 >> 172.21.15.150:0/2600139519 conn(0x7fecc80b78e0 msgr2=0x7fecc80b7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 -- 172.21.15.150:0/2600139519 shutdown_connections 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.082+0000 7feccf71f640 1 -- 172.21.15.150:0/2600139519 wait complete. 2024-04-25T14:06:48.083 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.083+0000 7feccf71f640 1 Processor -- start 2024-04-25T14:06:48.084 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.084+0000 7feccf71f640 1 -- start start 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc806d950 0x7fecc8162ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fecc80d44a0 0x7fecc8155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fecc81606b0 con 0x7fecc81602a0 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fecc806cd20 con 0x7fecc806d950 2024-04-25T14:06:48.086 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccf71f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fecc806dd60 con 0x7fecc80d44a0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47480/0 (socket says 172.21.15.150:47480) 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 -- 172.21.15.150:0/2072526081 learned_addr learned my addr 172.21.15.150:0/2072526081 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccccba640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fecc80d44a0 0x7fecc8155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccd4bb640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc806d950 0x7fecc8162ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 -- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fecc80d44a0 msgr2=0x7fecc8155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fecc80d44a0 0x7fecc8155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 -- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc806d950 msgr2=0x7fecc8162ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc806d950 0x7fecc8162ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 -- 172.21.15.150:0/2072526081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc80cb700 con 0x7fecc81602a0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7feccdcbc640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c700 tx=0x7fecc000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.085+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc0018070 con 0x7fecc81602a0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.086+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc000ce10 con 0x7fecc81602a0 2024-04-25T14:06:48.087 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.086+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc81609d0 con 0x7fecc81602a0 2024-04-25T14:06:48.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.086+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc000a860 con 0x7fecc81602a0 2024-04-25T14:06:48.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.086+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc8160f90 con 0x7fecc81602a0 2024-04-25T14:06:48.088 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.087+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecc8050740 con 0x7fecc81602a0 2024-04-25T14:06:48.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.087+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fecc001b070 con 0x7fecc81602a0 2024-04-25T14:06:48.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.088+0000 7fecbdffb640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 0x7fecac0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.088+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fecc0094960 con 0x7fecc81602a0 2024-04-25T14:06:48.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.088+0000 7feccd4bb640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 0x7fecac0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.089 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.088+0000 7feccd4bb640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 0x7fecac0769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fecc4034000 tx=0x7fecc402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.092 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.092+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fecc005ef60 con 0x7fecc81602a0 2024-04-25T14:06:48.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.301+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fecc80cb450 con 0x7fecc81602a0 2024-04-25T14:06:48.301 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.301+0000 7fecbdffb640 1 -- 172.21.15.150:0/2072526081 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7fecc005e860 con 0x7fecc81602a0 2024-04-25T14:06:48.302 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:48.302 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:48.302 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 msgr2=0x7fecac0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 0x7fecac0769c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fecc4034000 tx=0x7fecc402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 msgr2=0x7fecc80cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fecc000c700 tx=0x7fecc000cbc0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 shutdown_connections 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fecac0744b0 0x7fecac0769c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fecc81602a0 0x7fecc80cf080 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fecc80d44a0 0x7fecc8155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 --2- 172.21.15.150:0/2072526081 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fecc806d950 0x7fecc8162ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 >> 172.21.15.150:0/2072526081 conn(0x7fecc80b78e0 msgr2=0x7fecc815f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 shutdown_connections 2024-04-25T14:06:48.304 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.303+0000 7feccf71f640 1 -- 172.21.15.150:0/2072526081 wait complete. 2024-04-25T14:06:48.315 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json perf dump 2024-04-25T14:06:48.438 INFO:teuthology.orchestra.run.smithi150.stdout:{"AsyncMessenger::Worker-0":{"msgr_recv_messages":167,"msgr_send_messages":41,"msgr_recv_bytes":70697,"msgr_send_bytes":36340,"msgr_created_connections":9,"msgr_active_connections":5,"msgr_running_total_time":0.029497956,"msgr_running_send_time":0.008797377,"msgr_running_recv_time":0.011626490,"msgr_running_fast_dispatch_time":0.008219344,"msgr_send_messages_queue_lat":{"avgcount":40,"sum":0.006934539,"avgtime":0.000173363},"msgr_handle_ack_lat":{"avgcount":154,"sum":0.000036590,"avgtime":0.000000237},"msgr_recv_encrypted_bytes":1568,"msgr_send_encrypted_bytes":448},"AsyncMessenger::Worker-1":{"msgr_recv_messages":63,"msgr_send_messages":58,"msgr_recv_bytes":128216,"msgr_send_bytes":29754,"msgr_created_connections":5,"msgr_active_connections":4,"msgr_running_total_time":0.009260637,"msgr_running_send_time":0.003055567,"msgr_running_recv_time":0.005471662,"msgr_running_fast_dispatch_time":0.001114666,"msgr_send_messages_queue_lat":{"avgcount":58,"sum":0.005830765,"avgtime":0.000100530},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":124240,"msgr_send_encrypted_bytes":26336},"AsyncMessenger::Worker-2":{"msgr_recv_messages":28,"msgr_send_messages":56,"msgr_recv_bytes":14584,"msgr_send_bytes":149579,"msgr_created_connections":6,"msgr_active_connections":5,"msgr_running_total_time":0.006522715,"msgr_running_send_time":0.002662324,"msgr_running_recv_time":0.002725140,"msgr_running_fast_dispatch_time":0.000675086,"msgr_send_messages_queue_lat":{"avgcount":55,"sum":0.004440028,"avgtime":0.000080727},"msgr_handle_ack_lat":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"msgr_recv_encrypted_bytes":160,"msgr_send_encrypted_bytes":135248},"cct":{"total_workers":1,"unhealthy_workers":0},"finisher-MDSRank":{"queue_len":0,"complete_latency":{"avgcount":58,"sum":0.004854181,"avgtime":0.000083692}},"finisher-PurgeQueue":{"queue_len":0,"complete_latency":{"avgcount":4,"sum":0.000112594,"avgtime":0.000028148}},"mds":{"request":3,"reply":1,"reply_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"slow_reply":0,"forward":0,"dir_fetch_complete":12,"dir_fetch_keys":10,"dir_commit":0,"dir_split":0,"dir_merge":0,"inodes":11,"inodes_top":0,"inodes_bottom":11,"inodes_pin_tail":0,"inodes_pinned":11,"inodes_expired":0,"inodes_with_caps":2,"caps":2,"subtrees":2,"traverse":2,"traverse_hit":2,"traverse_forward":0,"traverse_discover":0,"traverse_dir_fetch":0,"traverse_remote_ino":0,"traverse_lock":0,"load_cent":0,"q":0,"exported":0,"exported_inodes":0,"imported":0,"imported_inodes":0,"openino_dir_fetch":0,"openino_backtrace_fetch":0,"openino_peer_discover":0,"root_rfiles":1,"root_rbytes":7,"root_rsnaps":0,"scrub_backtrace_fetch":0,"scrub_set_tag":0,"scrub_backtrace_repaired":0,"scrub_inotable_repaired":0,"scrub_dir_inodes":0,"scrub_dir_base_inodes":0,"scrub_dirfrag_rstats":0,"scrub_file_inodes":0,"handle_inode_file_caps":0,"ceph_cap_op_revoke":0,"ceph_cap_op_grant":1,"ceph_cap_op_trunc":0,"ceph_cap_op_flushsnap_ack":0,"ceph_cap_op_flush_ack":0,"handle_client_caps":2,"handle_client_caps_dirty":1,"handle_client_cap_release":0,"process_request_cap_release":1},"mds_cache":{"dir_update":0,"dir_update_receipt":0,"dir_try_discover":0,"dir_send_discover":0,"dir_handle_discover":0,"num_strays":0,"num_strays_delayed":0,"num_strays_enqueuing":0,"strays_created":0,"strays_enqueued":0,"strays_reintegrated":0,"strays_migrated":0,"num_recovering_processing":0,"num_recovering_enqueued":0,"num_recovering_prioritized":0,"recovery_started":0,"recovery_completed":0,"ireq_quiesce_path":0,"ireq_quiesce_inode":0,"ireq_enqueue_scrub":0,"ireq_exportdir":0,"ireq_flush":0,"ireq_fragmentdir":0,"ireq_fragstats":0,"ireq_inodestats":0},"mds_log":{"evlrg":0,"evadd":4,"evex":0,"evtrm":0,"ev":7,"evexg":0,"evexd":0,"segadd":0,"segex":0,"segtrm":0,"seg":2,"segmjr":2,"segexg":0,"segexd":0,"expos":4194304,"wrpos":4199960,"rdpos":4195244,"jlat":{"avgcount":4,"sum":0.008043456,"avgtime":0.002010864},"replayed":3},"mds_mem":{"ino":14,"ino+":14,"ino-":0,"dir":12,"dir+":12,"dir-":0,"dn":11,"dn+":11,"dn-":0,"cap":2,"cap+":2,"cap-":0,"rss":115476,"heap":248188},"mds_server":{"dispatch_client_request":2,"dispatch_server_request":0,"handle_client_request":3,"handle_client_session":7,"handle_peer_request":0,"req_create_latency":{"avgcount":1,"sum":0.002775347,"avgtime":0.002775347},"req_getattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_getfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_link_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookup_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookuphash_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupino_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupname_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupparent_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lookupsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_lssnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mkdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mknod_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_mksnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_open_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_readdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rename_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_renamesnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_snapdiff_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmdir_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmsnap_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_rmxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setdirlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setfilelock_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setlayout_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_setxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_symlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"req_unlink_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000},"cap_revoke_eviction":0,"cap_acquisition_throttle":0,"req_getvxattr_latency":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"mds_sessions":{"session_count":1,"session_add":1,"session_remove":0,"sessions_open":1,"sessions_stale":0,"total_load":1,"average_load":1,"avg_session_uptime":731,"mdthresh_evicted":0},"mempool":{"bloom_filter_bytes":0,"bloom_filter_items":0,"bluestore_alloc_bytes":0,"bluestore_alloc_items":0,"bluestore_cache_data_bytes":0,"bluestore_cache_data_items":0,"bluestore_cache_onode_bytes":0,"bluestore_cache_onode_items":0,"bluestore_cache_meta_bytes":0,"bluestore_cache_meta_items":0,"bluestore_cache_other_bytes":0,"bluestore_cache_other_items":0,"bluestore_cache_buffer_bytes":0,"bluestore_cache_buffer_items":0,"bluestore_extent_bytes":0,"bluestore_extent_items":0,"bluestore_blob_bytes":0,"bluestore_blob_items":0,"bluestore_shared_blob_bytes":0,"bluestore_shared_blob_items":0,"bluestore_inline_bl_bytes":0,"bluestore_inline_bl_items":0,"bluestore_fsck_bytes":0,"bluestore_fsck_items":0,"bluestore_txc_bytes":0,"bluestore_txc_items":0,"bluestore_writing_deferred_bytes":0,"bluestore_writing_deferred_items":0,"bluestore_writing_bytes":0,"bluestore_writing_items":0,"bluefs_bytes":0,"bluefs_items":0,"bluefs_file_reader_bytes":0,"bluefs_file_reader_items":0,"bluefs_file_writer_bytes":0,"bluefs_file_writer_items":0,"buffer_anon_bytes":237655,"buffer_anon_items":74,"buffer_meta_bytes":88,"buffer_meta_items":1,"osd_bytes":0,"osd_items":0,"osd_mapbl_bytes":0,"osd_mapbl_items":0,"osd_pglog_bytes":0,"osd_pglog_items":0,"osdmap_bytes":5848,"osdmap_items":95,"osdmap_mapping_bytes":0,"osdmap_mapping_items":0,"pgmap_bytes":0,"pgmap_items":0,"mds_co_bytes":56032,"mds_co_items":252,"unittest_1_bytes":0,"unittest_1_items":0,"unittest_2_bytes":0,"unittest_2_items":0},"objecter":{"op_active":0,"op_laggy":0,"op_send":57,"op_send_bytes":5926,"op_resend":0,"op_reply":57,"op_latency":{"avgcount":57,"sum":0.089998328,"avgtime":0.001578918},"op_inflight":0,"oplen_avg":{"avgcount":57,"sum":113},"op":57,"op_r":36,"op_w":21,"op_rmw":0,"op_pg":0,"osdop_stat":2,"osdop_create":0,"osdop_read":8,"osdop_write":4,"osdop_writefull":2,"osdop_writesame":0,"osdop_append":0,"osdop_zero":1,"osdop_truncate":0,"osdop_delete":5,"osdop_mapext":0,"osdop_sparse_read":0,"osdop_clonerange":0,"osdop_getxattr":24,"osdop_setxattr":0,"osdop_cmpxattr":0,"osdop_rmxattr":0,"osdop_resetxattrs":0,"osdop_call":0,"osdop_watch":0,"osdop_notify":0,"osdop_src_cmpxattr":0,"osdop_pgls":0,"osdop_pgls_filter":0,"osdop_other":0,"linger_active":0,"linger_send":0,"linger_resend":0,"linger_ping":0,"poolop_active":0,"poolop_send":0,"poolop_resend":0,"poolstat_active":0,"poolstat_send":0,"poolstat_resend":0,"statfs_active":0,"statfs_send":0,"statfs_resend":0,"command_active":0,"command_send":0,"command_resend":0,"map_epoch":62,"map_full":0,"map_inc":2,"osd_sessions":8,"osd_session_open":8,"osd_session_close":0,"osd_laggy":0,"omap_wr":15,"omap_rd":48,"omap_del":4},"oft":{"omap_total_objs":1,"omap_total_kv_pairs":1,"omap_total_updates":2,"omap_total_removes":1},"osd-slow-ops":{"slow_ops_count":0},"purge_queue":{"pq_executing_ops":0,"pq_executing_ops_high_water":0,"pq_executing":0,"pq_executing_high_water":0,"pq_executed_ops":0,"pq_executed":0,"pq_item_in_journal":0},"throttle-msgr_dispatch_throttler-mds":{"val":0,"max":104857600,"get_started":0,"get":253,"get_sum":186361,"get_or_fail_fail":0,"get_or_fail_success":253,"take":0,"take_sum":0,"put":253,"put_sum":186361,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_bytes":{"val":0,"max":104857600,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":6858,"put":57,"put_sum":6858,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-objecter_ops":{"val":0,"max":1024,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":57,"take_sum":57,"put":57,"put_sum":57,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle":{"val":0,"max":3758096384,"get_started":0,"get":0,"get_sum":0,"get_or_fail_fail":0,"get_or_fail_success":0,"take":0,"take_sum":0,"put":0,"put_sum":0,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}},"throttle-write_buf_throttle-0x5606921a4000":{"val":0,"max":3758096384,"get_started":0,"get":4,"get_sum":4460,"get_or_fail_fail":0,"get_or_fail_success":4,"take":0,"take_sum":0,"put":4,"put_sum":4460,"wait":{"avgcount":0,"sum":0.000000000,"avgtime":0.000000000}}} 2024-04-25T14:06:48.439 DEBUG:tasks.cephfs.filesystem:_json_asok output { "AsyncMessenger::Worker-0": { "msgr_active_connections": 5, "msgr_created_connections": 9, "msgr_handle_ack_lat": { "avgcount": 154, "avgtime": 2.37e-07, "sum": 3.659e-05 }, "msgr_recv_bytes": 70697, "msgr_recv_encrypted_bytes": 1568, "msgr_recv_messages": 167, "msgr_running_fast_dispatch_time": 0.008219344, "msgr_running_recv_time": 0.01162649, "msgr_running_send_time": 0.008797377, "msgr_running_total_time": 0.029497956, "msgr_send_bytes": 36340, "msgr_send_encrypted_bytes": 448, "msgr_send_messages": 41, "msgr_send_messages_queue_lat": { "avgcount": 40, "avgtime": 0.000173363, "sum": 0.006934539 } }, "AsyncMessenger::Worker-1": { "msgr_active_connections": 4, "msgr_created_connections": 5, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 128216, "msgr_recv_encrypted_bytes": 124240, "msgr_recv_messages": 63, "msgr_running_fast_dispatch_time": 0.001114666, "msgr_running_recv_time": 0.005471662, "msgr_running_send_time": 0.003055567, "msgr_running_total_time": 0.009260637, "msgr_send_bytes": 29754, "msgr_send_encrypted_bytes": 26336, "msgr_send_messages": 58, "msgr_send_messages_queue_lat": { "avgcount": 58, "avgtime": 0.00010053, "sum": 0.005830765 } }, "AsyncMessenger::Worker-2": { "msgr_active_connections": 5, "msgr_created_connections": 6, "msgr_handle_ack_lat": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "msgr_recv_bytes": 14584, "msgr_recv_encrypted_bytes": 160, "msgr_recv_messages": 28, "msgr_running_fast_dispatch_time": 0.000675086, "msgr_running_recv_time": 0.00272514, "msgr_running_send_time": 0.002662324, "msgr_running_total_time": 0.006522715, "msgr_send_bytes": 149579, "msgr_send_encrypted_bytes": 135248, "msgr_send_messages": 56, "msgr_send_messages_queue_lat": { "avgcount": 55, "avgtime": 8.0727e-05, "sum": 0.004440028 } }, "cct": { "total_workers": 1, "unhealthy_workers": 0 }, "finisher-MDSRank": { "complete_latency": { "avgcount": 58, "avgtime": 8.3692e-05, "sum": 0.004854181 }, "queue_len": 0 }, "finisher-PurgeQueue": { "complete_latency": { "avgcount": 4, "avgtime": 2.8148e-05, "sum": 0.000112594 }, "queue_len": 0 }, "mds": { "caps": 2, "ceph_cap_op_flush_ack": 0, "ceph_cap_op_flushsnap_ack": 0, "ceph_cap_op_grant": 1, "ceph_cap_op_revoke": 0, "ceph_cap_op_trunc": 0, "dir_commit": 0, "dir_fetch_complete": 12, "dir_fetch_keys": 10, "dir_merge": 0, "dir_split": 0, "exported": 0, "exported_inodes": 0, "forward": 0, "handle_client_cap_release": 0, "handle_client_caps": 2, "handle_client_caps_dirty": 1, "handle_inode_file_caps": 0, "imported": 0, "imported_inodes": 0, "inodes": 11, "inodes_bottom": 11, "inodes_expired": 0, "inodes_pin_tail": 0, "inodes_pinned": 11, "inodes_top": 0, "inodes_with_caps": 2, "load_cent": 0, "openino_backtrace_fetch": 0, "openino_dir_fetch": 0, "openino_peer_discover": 0, "process_request_cap_release": 1, "q": 0, "reply": 1, "reply_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "request": 3, "root_rbytes": 7, "root_rfiles": 1, "root_rsnaps": 0, "scrub_backtrace_fetch": 0, "scrub_backtrace_repaired": 0, "scrub_dir_base_inodes": 0, "scrub_dir_inodes": 0, "scrub_dirfrag_rstats": 0, "scrub_file_inodes": 0, "scrub_inotable_repaired": 0, "scrub_set_tag": 0, "slow_reply": 0, "subtrees": 2, "traverse": 2, "traverse_dir_fetch": 0, "traverse_discover": 0, "traverse_forward": 0, "traverse_hit": 2, "traverse_lock": 0, "traverse_remote_ino": 0 }, "mds_cache": { "dir_handle_discover": 0, "dir_send_discover": 0, "dir_try_discover": 0, "dir_update": 0, "dir_update_receipt": 0, "ireq_enqueue_scrub": 0, "ireq_exportdir": 0, "ireq_flush": 0, "ireq_fragmentdir": 0, "ireq_fragstats": 0, "ireq_inodestats": 0, "ireq_quiesce_inode": 0, "ireq_quiesce_path": 0, "num_recovering_enqueued": 0, "num_recovering_prioritized": 0, "num_recovering_processing": 0, "num_strays": 0, "num_strays_delayed": 0, "num_strays_enqueuing": 0, "recovery_completed": 0, "recovery_started": 0, "strays_created": 0, "strays_enqueued": 0, "strays_migrated": 0, "strays_reintegrated": 0 }, "mds_log": { "ev": 7, "evadd": 4, "evex": 0, "evexd": 0, "evexg": 0, "evlrg": 0, "evtrm": 0, "expos": 4194304, "jlat": { "avgcount": 4, "avgtime": 0.002010864, "sum": 0.008043456 }, "rdpos": 4195244, "replayed": 3, "seg": 2, "segadd": 0, "segex": 0, "segexd": 0, "segexg": 0, "segmjr": 2, "segtrm": 0, "wrpos": 4199960 }, "mds_mem": { "cap": 2, "cap+": 2, "cap-": 0, "dir": 12, "dir+": 12, "dir-": 0, "dn": 11, "dn+": 11, "dn-": 0, "heap": 248188, "ino": 14, "ino+": 14, "ino-": 0, "rss": 115476 }, "mds_server": { "cap_acquisition_throttle": 0, "cap_revoke_eviction": 0, "dispatch_client_request": 2, "dispatch_server_request": 0, "handle_client_request": 3, "handle_client_session": 7, "handle_peer_request": 0, "req_create_latency": { "avgcount": 1, "avgtime": 0.002775347, "sum": 0.002775347 }, "req_getattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_getvxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_link_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookup_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookuphash_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupino_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupname_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupparent_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lookupsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_lssnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mkdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mknod_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_mksnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_open_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_readdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rename_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_renamesnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmdir_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmsnap_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_rmxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setdirlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setfilelock_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setlayout_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_setxattr_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_snapdiff_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_symlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 }, "req_unlink_latency": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "mds_sessions": { "average_load": 1, "avg_session_uptime": 731, "mdthresh_evicted": 0, "session_add": 1, "session_count": 1, "session_remove": 0, "sessions_open": 1, "sessions_stale": 0, "total_load": 1 }, "mempool": { "bloom_filter_bytes": 0, "bloom_filter_items": 0, "bluefs_bytes": 0, "bluefs_file_reader_bytes": 0, "bluefs_file_reader_items": 0, "bluefs_file_writer_bytes": 0, "bluefs_file_writer_items": 0, "bluefs_items": 0, "bluestore_alloc_bytes": 0, "bluestore_alloc_items": 0, "bluestore_blob_bytes": 0, "bluestore_blob_items": 0, "bluestore_cache_buffer_bytes": 0, "bluestore_cache_buffer_items": 0, "bluestore_cache_data_bytes": 0, "bluestore_cache_data_items": 0, "bluestore_cache_meta_bytes": 0, "bluestore_cache_meta_items": 0, "bluestore_cache_onode_bytes": 0, "bluestore_cache_onode_items": 0, "bluestore_cache_other_bytes": 0, "bluestore_cache_other_items": 0, "bluestore_extent_bytes": 0, "bluestore_extent_items": 0, "bluestore_fsck_bytes": 0, "bluestore_fsck_items": 0, "bluestore_inline_bl_bytes": 0, "bluestore_inline_bl_items": 0, "bluestore_shared_blob_bytes": 0, "bluestore_shared_blob_items": 0, "bluestore_txc_bytes": 0, "bluestore_txc_items": 0, "bluestore_writing_bytes": 0, "bluestore_writing_deferred_bytes": 0, "bluestore_writing_deferred_items": 0, "bluestore_writing_items": 0, "buffer_anon_bytes": 237655, "buffer_anon_items": 74, "buffer_meta_bytes": 88, "buffer_meta_items": 1, "mds_co_bytes": 56032, "mds_co_items": 252, "osd_bytes": 0, "osd_items": 0, "osd_mapbl_bytes": 0, "osd_mapbl_items": 0, "osd_pglog_bytes": 0, "osd_pglog_items": 0, "osdmap_bytes": 5848, "osdmap_items": 95, "osdmap_mapping_bytes": 0, "osdmap_mapping_items": 0, "pgmap_bytes": 0, "pgmap_items": 0, "unittest_1_bytes": 0, "unittest_1_items": 0, "unittest_2_bytes": 0, "unittest_2_items": 0 }, "objecter": { "command_active": 0, "command_resend": 0, "command_send": 0, "linger_active": 0, "linger_ping": 0, "linger_resend": 0, "linger_send": 0, "map_epoch": 62, "map_full": 0, "map_inc": 2, "omap_del": 4, "omap_rd": 48, "omap_wr": 15, "op": 57, "op_Infinitylight": 0, "op_active": 0, "op_laggy": 0, "op_latency": { "avgcount": 57, "avgtime": 0.001578918, "sum": 0.089998328 }, "op_pg": 0, "op_r": 36, "op_reply": 57, "op_resend": 0, "op_rmw": 0, "op_send": 57, "op_send_bytes": 5926, "op_w": 21, "oplen_avg": { "avgcount": 57, "sum": 113 }, "osd_laggy": 0, "osd_session_close": 0, "osd_session_open": 8, "osd_sessions": 8, "osdop_append": 0, "osdop_call": 0, "osdop_clonerange": 0, "osdop_cmpxattr": 0, "osdop_create": 0, "osdop_delete": 5, "osdop_getxattr": 24, "osdop_mapext": 0, "osdop_notify": 0, "osdop_other": 0, "osdop_pgls": 0, "osdop_pgls_filter": 0, "osdop_read": 8, "osdop_resetxattrs": 0, "osdop_rmxattr": 0, "osdop_setxattr": 0, "osdop_sparse_read": 0, "osdop_src_cmpxattr": 0, "osdop_stat": 2, "osdop_truncate": 0, "osdop_watch": 0, "osdop_write": 4, "osdop_writefull": 2, "osdop_writesame": 0, "osdop_zero": 1, "poolop_active": 0, "poolop_resend": 0, "poolop_send": 0, "poolstat_active": 0, "poolstat_resend": 0, "poolstat_send": 0, "statfs_active": 0, "statfs_resend": 0, "statfs_send": 0 }, "oft": { "omap_total_kv_pairs": 1, "omap_total_objs": 1, "omap_total_removes": 1, "omap_total_updates": 2 }, "osd-slow-ops": { "slow_ops_count": 0 }, "purge_queue": { "pq_executed": 0, "pq_executed_ops": 0, "pq_executing": 0, "pq_executing_high_water": 0, "pq_executing_ops": 0, "pq_executing_ops_high_water": 0, "pq_item_in_journal": 0 }, "throttle-msgr_dispatch_throttler-mds": { "get": 253, "get_or_fail_fail": 0, "get_or_fail_success": 253, "get_started": 0, "get_sum": 186361, "max": 104857600, "put": 253, "put_sum": 186361, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_bytes": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 104857600, "put": 57, "put_sum": 6858, "take": 57, "take_sum": 6858, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-objecter_ops": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 1024, "put": 57, "put_sum": 57, "take": 57, "take_sum": 57, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle": { "get": 0, "get_or_fail_fail": 0, "get_or_fail_success": 0, "get_started": 0, "get_sum": 0, "max": 3758096384, "put": 0, "put_sum": 0, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } }, "throttle-write_buf_throttle-0x5606921a4000": { "get": 4, "get_or_fail_fail": 0, "get_or_fail_success": 4, "get_started": 0, "get_sum": 4460, "max": 3758096384, "put": 4, "put_sum": 4460, "take": 0, "take_sum": 0, "val": 0, "wait": { "avgcount": 0, "avgtime": 0.0, "sum": 0.0 } } } 2024-04-25T14:06:48.440 INFO:tasks.cephfs.test_openfiletable:omap_total_removes:1 2024-04-25T14:06:48.440 INFO:tasks.cephfs.test_openfiletable:omap_total_kv_pairs:1 2024-04-25T14:06:48.440 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-25T14:06:48.440 DEBUG:teuthology.orchestra.run.smithi150:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:06:48.472 DEBUG:teuthology.orchestra.run.smithi187:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-25T14:06:48.518 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-25T14:06:48.519 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:06:48.519 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:48.519 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:48.534 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:48.534 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:48.589 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:06:48.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.737+0000 7f70d6077640 1 Processor -- start 2024-04-25T14:06:48.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.737+0000 7f70d6077640 1 -- start start 2024-04-25T14:06:48.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70d6077640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70d00b5520 0x7f70d00b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70d6077640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006ec50 0x7f70d006f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.739 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70d6077640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70d00b5e70 con 0x7f70d00b2d10 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70d6077640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f70d0164ce0 con 0x7f70d00b5520 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70d6077640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0163f20 con 0x7f70d006ec50 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70d00b5520 0x7f70d00b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f70d00b5520 0x7f70d00b5930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47490/0 (socket says 172.21.15.150:47490) 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 -- 172.21.15.150:0/14190213 learned_addr learned my addr 172.21.15.150:0/14190213 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cffff640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006ec50 0x7f70d006f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 -- 172.21.15.150:0/14190213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006ec50 msgr2=0x7f70d006f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006ec50 0x7f70d006f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 -- 172.21.15.150:0/14190213 >> v1:172.21.15.187:6789/0 conn(0x7f70d00b2d10 legacy=0x7f70d00d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.740 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.739+0000 7f70cf7fe640 1 -- 172.21.15.150:0/14190213 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d0164200 con 0x7f70d00b5520 2024-04-25T14:06:48.741 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.740+0000 7f70cf7fe640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d00b5520 0x7f70d00b5930 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009870 tx=0x7f70b802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db00a7aaa4265d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.740+0000 7f70ceffd640 1 -- 172.21.15.150:0/14190213 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8037070 con 0x7f70d00b5520 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.740+0000 7f70ceffd640 1 -- 172.21.15.150:0/14190213 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b8032440 con 0x7f70d00b5520 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.740+0000 7f70ceffd640 1 -- 172.21.15.150:0/14190213 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8032810 con 0x7f70d00b5520 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 -- 172.21.15.150:0/14190213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d00b5520 msgr2=0x7f70d00b5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d00b5520 0x7f70d00b5930 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f70b8009870 tx=0x7f70b802a1c0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 -- 172.21.15.150:0/14190213 shutdown_connections 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d006ec50 0x7f70d006f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 --2- 172.21.15.150:0/14190213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d00b5520 0x7f70d00b5930 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 -- 172.21.15.150:0/14190213 >> 172.21.15.150:0/14190213 conn(0x7f70d00d0520 msgr2=0x7f70d00d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 -- 172.21.15.150:0/14190213 shutdown_connections 2024-04-25T14:06:48.742 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.741+0000 7f70d6077640 1 -- 172.21.15.150:0/14190213 wait complete. 2024-04-25T14:06:48.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.742+0000 7f70d6077640 1 Processor -- start 2024-04-25T14:06:48.743 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.743+0000 7f70d6077640 1 -- start start 2024-04-25T14:06:48.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00b2d10 0x7f70d01ea7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.745 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d00b5520 0x7f70d0161880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01604c0 con 0x7f70d006ec50 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f70d01647c0 con 0x7f70d00b2d10 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70d6077640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f70d0160640 con 0x7f70d00b5520 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47498/0 (socket says 172.21.15.150:47498) 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 -- 172.21.15.150:0/1380857680 learned_addr learned my addr 172.21.15.150:0/1380857680 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70cf7fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00b2d10 0x7f70d01ea7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70cffff640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d00b5520 0x7f70d0161880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 -- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d00b5520 msgr2=0x7f70d0161880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d00b5520 0x7f70d0161880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 -- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00b2d10 msgr2=0x7f70d01ea7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00b2d10 0x7f70d01ea7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.746 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.744+0000 7f70c77fe640 1 -- 172.21.15.150:0/1380857680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d01e9d10 con 0x7f70d006ec50 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70c77fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c3f0 tx=0x7f70c000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c00052d0 con 0x7f70d006ec50 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70c0005490 con 0x7f70d006ec50 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d01f6700 con 0x7f70d006ec50 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0007260 con 0x7f70d006ec50 2024-04-25T14:06:48.747 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.745+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d01e9fb0 con 0x7f70d006ec50 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.746+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d0050740 con 0x7f70d006ec50 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.746+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f70c0007400 con 0x7f70d006ec50 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.747+0000 7f70ccff9640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 0x7f70b00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.747+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f70c0094840 con 0x7f70d006ec50 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.747+0000 7f70cf7fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 0x7f70b00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:48.748 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.747+0000 7f70cf7fe640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 0x7f70b00769f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:48.751 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.751+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f70d00b5e70 con 0x7f70d006ec50 2024-04-25T14:06:48.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.927+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f70d01ebf30 con 0x7f70d006ec50 2024-04-25T14:06:48.928 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.928+0000 7f70ccff9640 1 -- 172.21.15.150:0/1380857680 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+17111 (secure 0 0 0) 0x7f70c005e750 con 0x7f70d006ec50 2024-04-25T14:06:48.928 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:48.928 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":62,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:51.296371+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/133937444":"2024-04-26T14:04:37.253920+0000","172.21.15.187:6833/133937444":"2024-04-26T14:04:37.253920+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:06:48.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 msgr2=0x7f70b00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 0x7f70b00769f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f70b8034000 tx=0x7f70b802e000 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 msgr2=0x7f70d00b7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f70c000c3f0 tx=0x7f70c000c8b0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 shutdown_connections 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f70b00744e0 0x7f70b00769f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f70d00b5520 0x7f70d0161880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f70d00b2d10 0x7f70d01ea7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 --2- 172.21.15.150:0/1380857680 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f70d006ec50 0x7f70d00b7750 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 >> 172.21.15.150:0/1380857680 conn(0x7f70d00d0520 msgr2=0x7f70d00ce630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 shutdown_connections 2024-04-25T14:06:48.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:48.930+0000 7f70d6077640 1 -- 172.21.15.150:0/1380857680 wait complete. 2024-04-25T14:06:48.942 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:04:27.836897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:04:28.513769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:06:48.943 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:06:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.054+0000 7fe51479d640 1 Processor -- start 2024-04-25T14:06:49.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.054+0000 7fe51479d640 1 -- start start 2024-04-25T14:06:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe51479d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe50c0ce510 0x7fe50c071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.057 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe51479d640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50c071eb0 con 0x7fe50c071aa0 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe51479d640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe50c162240 con 0x7fe50c0b3160 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe51479d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe50c162520 con 0x7fe50c0ce510 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe512d3a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe50c071aa0 0x7fe50c0d25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37428/0 (socket says 172.21.15.150:37428) 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.055+0000 7fe512d3a640 1 -- 172.21.15.150:0/3318468939 learned_addr learned my addr 172.21.15.150:0/3318468939 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511d38640 1 --2- 172.21.15.150:0/3318468939 >> v2:172.21.15.187:3300/0 conn(0x7fe50c0ce510 0x7fe50c071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1417702327 0 0) 0x7fe504001490 con 0x7fe50c071aa0 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4f8003520 con 0x7fe50c071aa0 2024-04-25T14:06:49.058 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4181788701 0 0) 0x7fe4f8003520 con 0x7fe50c071aa0 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 >> v2:172.21.15.187:3300/0 conn(0x7fe50c0ce510 msgr2=0x7fe50c071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 --2- 172.21.15.150:0/3318468939 >> v2:172.21.15.187:3300/0 conn(0x7fe50c0ce510 0x7fe50c071560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 >> v1:172.21.15.187:6789/0 conn(0x7fe50c0b3160 legacy=0x7fe50c15e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c162800 con 0x7fe50c071aa0 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fe504001bc0 con 0x7fe50c071aa0 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 >> v1:172.21.15.150:6789/0 conn(0x7fe50c071aa0 legacy=0x7fe50c0d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe4f8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4f8004800 0x7fe4f8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4f8005270 0x7fe4f800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c162520 con 0x7fe4f8003a00 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe4f800c190 con 0x7fe4f8004800 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe4f800c390 con 0x7fe4f8005270 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512539640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe4f8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.059 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe511d38640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4f8004800 0x7fe4f8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512d3a640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4f8005270 0x7fe4f800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512539640 1 -- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4f8005270 msgr2=0x7fe4f800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512539640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe4f8005270 0x7fe4f800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512539640 1 -- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4f8004800 msgr2=0x7fe4f8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.056+0000 7fe512539640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe4f8004800 0x7fe4f8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe512d3a640 1 -- 172.21.15.150:0/3318468939 reap_dead start 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe512539640 1 -- 172.21.15.150:0/3318468939 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4f8003520 con 0x7fe4f8003a00 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe512539640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe4f8003e70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe508001750 tx=0x7fe4f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d845845c17f265 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508001370 con 0x7fe4f8003a00 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5080052d0 con 0x7fe4f8003a00 2024-04-25T14:06:49.060 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.057+0000 7fe511537640 1 -- 172.21.15.150:0/3318468939 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe50802abc0 con 0x7fe4f8003a00 2024-04-25T14:06:49.063 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 -- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 msgr2=0x7fe4f8003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe4f8003e70 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fe508001750 tx=0x7fe4f8005830 comp rx=0 tx=0).stop 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 -- 172.21.15.150:0/3318468939 shutdown_connections 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 --2- 172.21.15.150:0/3318468939 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe4f8003e70 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 -- 172.21.15.150:0/3318468939 >> 172.21.15.150:0/3318468939 conn(0x7fe50c0d1170 msgr2=0x7fe50c0d3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 -- 172.21.15.150:0/3318468939 shutdown_connections 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.058+0000 7fe51479d640 1 -- 172.21.15.150:0/3318468939 wait complete. 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.060+0000 7fe51479d640 1 Processor -- start 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.060+0000 7fe51479d640 1 -- start start 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 0x7fe50c0cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 0x7fe50c14d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.064 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c161f50 con 0x7fe4f8003a00 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe50c161dd0 con 0x7fe50c15e0b0 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe51479d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe50c0b3160 con 0x7fe50c0cd200 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe512539640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe512539640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47512/0 (socket says 172.21.15.150:47512) 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.061+0000 7fe512539640 1 -- 172.21.15.150:0/1626976376 learned_addr learned my addr 172.21.15.150:0/1626976376 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512d3a640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 0x7fe50c14d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe511d38640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 0x7fe50c0cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 -- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 msgr2=0x7fe50c14d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 0x7fe50c14d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 -- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 msgr2=0x7fe50c0cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 0x7fe50c0cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 -- 172.21.15.150:0/1626976376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe50c0b2480 con 0x7fe4f8003a00 2024-04-25T14:06:49.065 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512d3a640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 0x7fe50c14d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe511d38640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 0x7fe50c0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe512539640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe508001870 tx=0x7fe508001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508037040 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe508005480 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe508001370 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe50c0cd610 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.062+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe50c15ee10 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.063+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50c0d1db0 con 0x7fe4f8003a00 2024-04-25T14:06:49.066 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.064+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe508043020 con 0x7fe4f8003a00 2024-04-25T14:06:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.064+0000 7fe502ffd640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 0x7fe4f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.064+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fe5080b49f0 con 0x7fe4f8003a00 2024-04-25T14:06:49.067 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.064+0000 7fe511d38640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 0x7fe4f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.068 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.065+0000 7fe511d38640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 0x7fe4f00769f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe50c15b090 tx=0x7fe4fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.069 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.069+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe50804a020 con 0x7fe4f8003a00 2024-04-25T14:06:49.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.244+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe50c0d3090 con 0x7fe4f8003a00 2024-04-25T14:06:49.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.245+0000 7fe502ffd640 1 -- 172.21.15.150:0/1626976376 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (secure 0 0 0) 0x7fe50c0cd610 con 0x7fe4f8003a00 2024-04-25T14:06:49.245 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:06:49.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 msgr2=0x7fe4f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 0x7fe4f00769f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe50c15b090 tx=0x7fe4fc008040 comp rx=0 tx=0).stop 2024-04-25T14:06:49.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 msgr2=0x7fe50c0b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe508001870 tx=0x7fe508001120 comp rx=0 tx=0).stop 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 shutdown_connections 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe4f00744e0 0x7fe4f00769f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe50c0cd200 0x7fe50c14d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe50c15e0b0 0x7fe50c0cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 --2- 172.21.15.150:0/1626976376 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe4f8003a00 0x7fe50c0b1f40 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 >> 172.21.15.150:0/1626976376 conn(0x7fe50c0d1170 msgr2=0x7fe50c0b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 shutdown_connections 2024-04-25T14:06:49.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.247+0000 7fe51479d640 1 -- 172.21.15.150:0/1626976376 wait complete. 2024-04-25T14:06:49.258 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:06:49.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.369+0000 7f539fb00640 1 Processor -- start 2024-04-25T14:06:49.369 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.369+0000 7f539fb00640 1 -- start start 2024-04-25T14:06:49.371 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 0x7f539815e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53980ce390 0x7f53980b5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53980b5a60 0x7f5398072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f53980d2080 con 0x7f53980ce390 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f53980b5e70 con 0x7f5398161bd0 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539fb00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53980cebe0 con 0x7f53980b5a60 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.370+0000 7f539d09b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53980ce390 0x7f53980b5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f53980ce390 0x7f53980b5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47518/0 (socket says 172.21.15.150:47518) 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 -- 172.21.15.150:0/3043239468 learned_addr learned my addr 172.21.15.150:0/3043239468 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539e09d640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53980b5a60 0x7f5398072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.372 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d89c640 1 --2- 172.21.15.150:0/3043239468 >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 0x7f539815e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 -- 172.21.15.150:0/3043239468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53980b5a60 msgr2=0x7f5398072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53980b5a60 0x7f5398072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 -- 172.21.15.150:0/3043239468 >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 msgr2=0x7f539815e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 --2- 172.21.15.150:0/3043239468 >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 0x7f539815e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d09b640 1 -- 172.21.15.150:0/3043239468 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53980ceec0 con 0x7f53980ce390 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.371+0000 7f539d89c640 1 --2- 172.21.15.150:0/3043239468 >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 0x7f539815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539d09b640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980ce390 0x7f53980b5520 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f53980b6310 tx=0x7f538802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c36099c4ec2646 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539c89a640 1 -- 172.21.15.150:0/3043239468 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388037070 con 0x7f53980ce390 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539c89a640 1 -- 172.21.15.150:0/3043239468 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5388032440 con 0x7f53980ce390 2024-04-25T14:06:49.373 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539c89a640 1 -- 172.21.15.150:0/3043239468 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388032810 con 0x7f53980ce390 2024-04-25T14:06:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 -- 172.21.15.150:0/3043239468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980ce390 msgr2=0x7f53980b5520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980ce390 0x7f53980b5520 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f53980b6310 tx=0x7f538802a140 comp rx=0 tx=0).stop 2024-04-25T14:06:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 -- 172.21.15.150:0/3043239468 shutdown_connections 2024-04-25T14:06:49.374 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f53980b5a60 0x7f5398072fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 --2- 172.21.15.150:0/3043239468 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980ce390 0x7f53980b5520 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 --2- 172.21.15.150:0/3043239468 >> v2:172.21.15.187:3300/0 conn(0x7f5398161bd0 0x7f539815e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.372+0000 7f539fb00640 1 -- 172.21.15.150:0/3043239468 >> 172.21.15.150:0/3043239468 conn(0x7f53980d0520 msgr2=0x7f53980d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.373+0000 7f539fb00640 1 -- 172.21.15.150:0/3043239468 shutdown_connections 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.373+0000 7f539fb00640 1 -- 172.21.15.150:0/3043239468 wait complete. 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.374+0000 7f539fb00640 1 Processor -- start 2024-04-25T14:06:49.375 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.374+0000 7f539fb00640 1 -- start start 2024-04-25T14:06:49.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53980ce390 0x7f53980cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5398161bd0 0x7f5398160880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.377 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f53980b6d80 con 0x7f53980b5a60 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f53980d22d0 con 0x7f53980ce390 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f53980b6f00 con 0x7f5398161bd0 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47522/0 (socket says 172.21.15.150:47522) 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 -- 172.21.15.150:0/1481632590 learned_addr learned my addr 172.21.15.150:0/1481632590 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539e09d640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5398161bd0 0x7f5398160880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d09b640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53980ce390 0x7f53980cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 -- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5398161bd0 msgr2=0x7f5398160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5398161bd0 0x7f5398160880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 -- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53980ce390 msgr2=0x7f53980cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53980ce390 0x7f53980cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 -- 172.21.15.150:0/1481632590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53980cd0e0 con 0x7f53980b5a60 2024-04-25T14:06:49.378 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539d89c640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f539400d7d0 tx=0x7f539400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539400b6c0 con 0x7f53980b5a60 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5394019070 con 0x7f53980b5a60 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53980c7610 con 0x7f53980b5a60 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.376+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539400baa0 con 0x7f53980b5a60 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.377+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5398157740 con 0x7f53980b5a60 2024-04-25T14:06:49.379 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.377+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5398050740 con 0x7f53980b5a60 2024-04-25T14:06:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.378+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f539400bc40 con 0x7f53980b5a60 2024-04-25T14:06:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.379+0000 7f538e7fc640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.379+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f5394095830 con 0x7f53980b5a60 2024-04-25T14:06:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.379+0000 7f539d09b640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.380 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.379+0000 7f539d09b640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 0x7f537c0769c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5388034000 tx=0x7f538802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.383 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.383+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f539405fda0 con 0x7f53980b5a60 2024-04-25T14:06:49.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.558+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f53980d1c30 con 0x7f53980b5a60 2024-04-25T14:06:49.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.558+0000 7f538e7fc640 1 -- 172.21.15.150:0/1481632590 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) ==== 104+0+11 (secure 0 0 0) 0x7f539405f6a0 con 0x7f53980b5a60 2024-04-25T14:06:49.559 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 msgr2=0x7f537c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 0x7f537c0769c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f5388034000 tx=0x7f538802e000 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 msgr2=0x7f5398157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f539400d7d0 tx=0x7f539400dc90 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 shutdown_connections 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f537c0744b0 0x7f537c0769c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f5398161bd0 0x7f5398160880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f53980ce390 0x7f53980cc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 --2- 172.21.15.150:0/1481632590 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f53980b5a60 0x7f5398157200 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 >> 172.21.15.150:0/1481632590 conn(0x7f53980d0520 msgr2=0x7f53980b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 shutdown_connections 2024-04-25T14:06:49.562 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.560+0000 7f539fb00640 1 -- 172.21.15.150:0/1481632590 wait complete. 2024-04-25T14:06:49.571 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:06:49.682 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.682+0000 7fd69b59e640 1 Processor -- start 2024-04-25T14:06:49.683 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.683+0000 7fd69b59e640 1 -- start start 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd69b59e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd69415dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd69b59e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd694073a10 0x7fd694073e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd69b59e640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd694074360 con 0x7fd69415ac80 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd69b59e640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fd69415da10 con 0x7fd694073a10 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd69b59e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6940b0bd0 con 0x7fd6940b07c0 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd694073a10 0x7fd694073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fd694073a10 0x7fd694073e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47538/0 (socket says 172.21.15.150:47538) 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 -- 172.21.15.150:0/949774108 learned_addr learned my addr 172.21.15.150:0/949774108 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd698b39640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd69415dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 -- 172.21.15.150:0/949774108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 msgr2=0x7fd69415dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd69415dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 -- 172.21.15.150:0/949774108 >> v1:172.21.15.187:6789/0 conn(0x7fd69415ac80 legacy=0x7fd6940b67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.685 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.684+0000 7fd699b3b640 1 -- 172.21.15.150:0/949774108 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6940b0ea0 con 0x7fd694073a10 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.685+0000 7fd699b3b640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd694073e20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00bb10 tx=0x7fd68c02c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b5cb834bb117093 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.685+0000 7fd68bfff640 1 -- 172.21.15.150:0/949774108 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c036020 con 0x7fd694073a10 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.685+0000 7fd68bfff640 1 -- 172.21.15.150:0/949774108 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd68c032820 con 0x7fd694073a10 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.685+0000 7fd68bfff640 1 -- 172.21.15.150:0/949774108 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c032bf0 con 0x7fd694073a10 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 -- 172.21.15.150:0/949774108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 msgr2=0x7fd694073e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd694073e20 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00bb10 tx=0x7fd68c02c920 comp rx=0 tx=0).stop 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 -- 172.21.15.150:0/949774108 shutdown_connections 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd694073e20 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 --2- 172.21.15.150:0/949774108 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd69415dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 -- 172.21.15.150:0/949774108 >> 172.21.15.150:0/949774108 conn(0x7fd6940d1ee0 msgr2=0x7fd694073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 -- 172.21.15.150:0/949774108 shutdown_connections 2024-04-25T14:06:49.687 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.686+0000 7fd69b59e640 1 -- 172.21.15.150:0/949774108 wait complete. 2024-04-25T14:06:49.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.688+0000 7fd69b59e640 1 Processor -- start 2024-04-25T14:06:49.688 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.688+0000 7fd69b59e640 1 -- start start 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd6941439e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd69415ac80 0x7fd69415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd69415e3e0 con 0x7fd694073a10 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd69415d6a0 con 0x7fd69415ac80 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69b59e640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd6940b4c30 con 0x7fd6940b07c0 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69933a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69933a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47554/0 (socket says 172.21.15.150:47554) 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd69933a640 1 -- 172.21.15.150:0/3481123231 learned_addr learned my addr 172.21.15.150:0/3481123231 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd699b3b640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd69415ac80 0x7fd69415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.689+0000 7fd698b39640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd6941439e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.691 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 -- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 msgr2=0x7fd6941439e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd6941439e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 -- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd69415ac80 msgr2=0x7fd69415c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd69415ac80 0x7fd69415c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 -- 172.21.15.150:0/3481123231 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6941e3bd0 con 0x7fd694073a10 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69933a640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c730 tx=0x7fd69000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690016020 con 0x7fd694073a10 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd690006d50 con 0x7fd694073a10 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd694146400 con 0x7fd694073a10 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd690003bb0 con 0x7fd694073a10 2024-04-25T14:06:49.692 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.690+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6941469c0 con 0x7fd694073a10 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.691+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd694050740 con 0x7fd694073a10 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.692+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd69000a850 con 0x7fd694073a10 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.692+0000 7fd689ffb640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 0x7fd6780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.692+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fd690094e80 con 0x7fd694073a10 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.692+0000 7fd698b39640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 0x7fd6780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:49.693 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.693+0000 7fd698b39640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 0x7fd6780769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd69406d6c0 tx=0x7fd684008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:49.697 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.697+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd690020020 con 0x7fd694073a10 2024-04-25T14:06:49.874 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.874+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6941570e0 con 0x7fd694073a10 2024-04-25T14:06:49.875 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.875+0000 7fd689ffb640 1 -- 172.21.15.150:0/3481123231 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) ==== 100+0+11 (secure 0 0 0) 0x7fd694074360 con 0x7fd694073a10 2024-04-25T14:06:49.875 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:06:49.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 msgr2=0x7fd6780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 0x7fd6780769c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fd69406d6c0 tx=0x7fd684008040 comp rx=0 tx=0).stop 2024-04-25T14:06:49.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 msgr2=0x7fd6941e3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:49.877 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd69000c730 tx=0x7fd69000cbf0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 shutdown_connections 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd6780744b0 0x7fd6780769c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd69415ac80 0x7fd69415c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd6940b07c0 0x7fd6941439e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 --2- 172.21.15.150:0/3481123231 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd694073a10 0x7fd6941e3690 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 >> 172.21.15.150:0/3481123231 conn(0x7fd6940d1ee0 msgr2=0x7fd69406f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 shutdown_connections 2024-04-25T14:06:49.878 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.877+0000 7fd69b59e640 1 -- 172.21.15.150:0/3481123231 wait complete. 2024-04-25T14:06:49.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T14:06:50.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.996+0000 7f413d39d640 1 Processor -- start 2024-04-25T14:06:50.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.997+0000 7f413d39d640 1 -- start start 2024-04-25T14:06:50.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41380b3160 0x7f413815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41380ce510 0x7f4138071560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f4138071aa0 0x7f4138070170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4138071eb0 con 0x7f41380ce510 2024-04-25T14:06:50.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f4138162240 con 0x7f4138071aa0 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f413d39d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4138162520 con 0x7f41380b3160 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41380ce510 0x7f4138071560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41380ce510 0x7f4138071560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47570/0 (socket says 172.21.15.150:47570) 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 -- 172.21.15.150:0/2363686804 learned_addr learned my addr 172.21.15.150:0/2363686804 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f4136ffd640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41380b3160 0x7f413815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41377fe640 1 --2- 172.21.15.150:0/2363686804 >> v2:172.21.15.187:3300/0 conn(0x7f4138071aa0 0x7f4138070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 -- 172.21.15.150:0/2363686804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41380b3160 msgr2=0x7f413815e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41380b3160 0x7f413815e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 -- 172.21.15.150:0/2363686804 >> v2:172.21.15.187:3300/0 conn(0x7f4138071aa0 msgr2=0x7f4138070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 --2- 172.21.15.150:0/2363686804 >> v2:172.21.15.187:3300/0 conn(0x7f4138071aa0 0x7f4138070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.998+0000 7f41367fc640 1 -- 172.21.15.150:0/2363686804 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4138162800 con 0x7f41380ce510 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.999+0000 7f41367fc640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41380ce510 0x7f4138071560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f41380ce100 tx=0x7f412c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4170fc9dbbeee7c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.999+0000 7f4135ffb640 1 -- 172.21.15.150:0/2363686804 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c037070 con 0x7f41380ce510 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.999+0000 7f4135ffb640 1 -- 172.21.15.150:0/2363686804 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412c032440 con 0x7f41380ce510 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:49.999+0000 7f4135ffb640 1 -- 172.21.15.150:0/2363686804 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c032810 con 0x7f41380ce510 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 -- 172.21.15.150:0/2363686804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41380ce510 msgr2=0x7f4138071560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41380ce510 0x7f4138071560 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f41380ce100 tx=0x7f412c02a1e0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 -- 172.21.15.150:0/2363686804 shutdown_connections 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 --2- 172.21.15.150:0/2363686804 >> v2:172.21.15.187:3300/0 conn(0x7f4138071aa0 0x7f4138070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41380ce510 0x7f4138071560 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 --2- 172.21.15.150:0/2363686804 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41380b3160 0x7f413815e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 -- 172.21.15.150:0/2363686804 >> 172.21.15.150:0/2363686804 conn(0x7f41380d1170 msgr2=0x7f41380d3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 -- 172.21.15.150:0/2363686804 shutdown_connections 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.000+0000 7f413d39d640 1 -- 172.21.15.150:0/2363686804 wait complete. 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.002+0000 7f413d39d640 1 Processor -- start 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.002+0000 7f413d39d640 1 -- start start 2024-04-25T14:06:50.020 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4138071aa0 0x7f41380cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41380ce510 0x7f413814a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4138161dd0 con 0x7f413814a720 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f413815e5f0 con 0x7f41380ce510 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.003+0000 7f413d39d640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4138161f50 con 0x7f4138071aa0 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47576/0 (socket says 172.21.15.150:47576) 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 -- 172.21.15.150:0/1308128981 learned_addr learned my addr 172.21.15.150:0/1308128981 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f4136ffd640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4138071aa0 0x7f41380cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41367fc640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41380ce510 0x7f413814a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 -- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4138071aa0 msgr2=0x7f41380cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4138071aa0 0x7f41380cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 -- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41380ce510 msgr2=0x7f413814a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41380ce510 0x7f413814a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 -- 172.21.15.150:0/1308128981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41380c91e0 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f41377fe640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f413000e7f0 tx=0x7f413000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130017070 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f413000c730 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413000cb40 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41380c94c0 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.004+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413814af80 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.005+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413815ebe0 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.006+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f413001a070 con 0x7f413814a720 2024-04-25T14:06:50.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.006+0000 7f411f7fe640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 0x7f41240769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.007+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f41300953a0 con 0x7f413814a720 2024-04-25T14:06:50.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.007+0000 7f4136ffd640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 0x7f41240769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.007+0000 7f4136ffd640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 0x7f41240769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4128011340 tx=0x7f412800b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.011+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f413005f3e0 con 0x7f413814a720 2024-04-25T14:06:50.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.197+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4138050740 con 0x7f413814a720 2024-04-25T14:06:50.197 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.197+0000 7f411f7fe640 1 -- 172.21.15.150:0/1308128981 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v62) ==== 80+0+122 (secure 0 0 0) 0x7f41300636e0 con 0x7f413814a720 2024-04-25T14:06:50.198 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.187:6835/133937444 2024-04-26T14:04:37.253920+0000 2024-04-25T14:06:50.198 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.187:6833/133937444 2024-04-26T14:04:37.253920+0000 2024-04-25T14:06:50.198 INFO:teuthology.orchestra.run.smithi150.stderr:listed 2 entries 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 msgr2=0x7f41240769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 0x7f41240769f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4128011340 tx=0x7f412800b040 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 msgr2=0x7f4138160490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f413000e7f0 tx=0x7f413000ecb0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 shutdown_connections 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41240744e0 0x7f41240769f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f413814a720 0x7f4138160490 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41380ce510 0x7f413814a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 --2- 172.21.15.150:0/1308128981 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4138071aa0 0x7f41380cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 >> 172.21.15.150:0/1308128981 conn(0x7f41380d1170 msgr2=0x7f41380b8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 shutdown_connections 2024-04-25T14:06:50.200 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.199+0000 7f413d39d640 1 -- 172.21.15.150:0/1308128981 wait complete. 2024-04-25T14:06:50.210 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T14:06:50.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:50.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T14:06:50.237 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:06:50.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:50.238 DEBUG:teuthology.orchestra.run.smithi150:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T14:06:50.293 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:06:50.293 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:06:50.293 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:50.312 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:06:50.312 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:06:50.312 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:50.312 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:50.348 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:06:50.348 INFO:tasks.cephfs.mount:Terminating background process 2024-04-25T14:06:50.348 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:06:50.348 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:50.370 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:06:50.371 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.495+0000 7fb40dc05640 1 Processor -- start 2024-04-25T14:06:50.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.495+0000 7fb40dc05640 1 -- start start 2024-04-25T14:06:50.497 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb40dc05640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080cebe0 0x7fb4081648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb40dc05640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4080b45f0 0x7fb4080b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb40dc05640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb408160810 con 0x7fb4080d1ae0 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb40dc05640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb4080d0300 con 0x7fb4080b45f0 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb40dc05640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb408164e30 con 0x7fb4080cebe0 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.496+0000 7fb406ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4080b45f0 0x7fb4080b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb407fff640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fb4080d1ae0 0x7fb4080d1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37442/0 (socket says 172.21.15.150:37442) 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb407fff640 1 -- 172.21.15.150:0/802533445 learned_addr learned my addr 172.21.15.150:0/802533445 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb4077fe640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080cebe0 0x7fb4081648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb406ffd640 1 -- 172.21.15.150:0/802533445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080cebe0 msgr2=0x7fb4081648f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb406ffd640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080cebe0 0x7fb4081648f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb406ffd640 1 -- 172.21.15.150:0/802533445 >> v1:172.21.15.150:6789/0 conn(0x7fb4080d1ae0 legacy=0x7fb4080d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.498 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb4067fc640 1 -- 172.21.15.150:0/802533445 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794959694 0 0) 0x7fb408160810 con 0x7fb4080d1ae0 2024-04-25T14:06:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.497+0000 7fb406ffd640 1 -- 172.21.15.150:0/802533445 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4080b4f40 con 0x7fb4080b45f0 2024-04-25T14:06:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb406ffd640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4080b4a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0097d0 tx=0x7fb3fc02a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=233d65531a996a70 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb4067fc640 1 -- 172.21.15.150:0/802533445 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc037070 con 0x7fb4080b45f0 2024-04-25T14:06:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb4067fc640 1 -- 172.21.15.150:0/802533445 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3fc032440 con 0x7fb4080b45f0 2024-04-25T14:06:50.499 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb4067fc640 1 -- 172.21.15.150:0/802533445 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc032810 con 0x7fb4080b45f0 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 -- 172.21.15.150:0/802533445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 msgr2=0x7fb4080b4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4080b4a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc0097d0 tx=0x7fb3fc02a1c0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 -- 172.21.15.150:0/802533445 shutdown_connections 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4080b4a00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 --2- 172.21.15.150:0/802533445 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080cebe0 0x7fb4081648f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 -- 172.21.15.150:0/802533445 >> 172.21.15.150:0/802533445 conn(0x7fb4080b76d0 msgr2=0x7fb4080b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 -- 172.21.15.150:0/802533445 shutdown_connections 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.498+0000 7fb40dc05640 1 -- 172.21.15.150:0/802533445 wait complete. 2024-04-25T14:06:50.500 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.500+0000 7fb40dc05640 1 Processor -- start 2024-04-25T14:06:50.501 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.500+0000 7fb40dc05640 1 -- start start 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 0x7fb4081e9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 0x7fb4081eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080cfe10 con 0x7fb4080b45f0 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb4080b4160 con 0x7fb4080cebe0 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb4080cfb30 con 0x7fb4080d1ae0 2024-04-25T14:06:50.503 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47600/0 (socket says 172.21.15.150:47600) 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 -- 172.21.15.150:0/1509767909 learned_addr learned my addr 172.21.15.150:0/1509767909 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb407fff640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 0x7fb4081eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb406ffd640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 0x7fb4081e9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 -- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 msgr2=0x7fb4081eb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 0x7fb4081eb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 -- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 msgr2=0x7fb4081e9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 0x7fb4081e9cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 -- 172.21.15.150:0/1509767909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4081f6df0 con 0x7fb4080b45f0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb407fff640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 0x7fb4081eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb406ffd640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 0x7fb4081e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb4077fe640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d800 tx=0x7fb3f800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8005990 con 0x7fb4080b45f0 2024-04-25T14:06:50.504 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f8005b30 con 0x7fb4080b45f0 2024-04-25T14:06:50.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.502+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081f70d0 con 0x7fb4080b45f0 2024-04-25T14:06:50.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.503+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8005d20 con 0x7fb4080b45f0 2024-04-25T14:06:50.505 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.503+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb408159f20 con 0x7fb4080b45f0 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.504+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4081f75f0 con 0x7fb4080b45f0 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.504+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb3f801b020 con 0x7fb4080b45f0 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.505+0000 7fb3effff640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 0x7fb3f40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.505+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fb3f8095160 con 0x7fb4080b45f0 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.505+0000 7fb406ffd640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 0x7fb3f40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.506 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.505+0000 7fb406ffd640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 0x7fb3f40769c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc034000 tx=0x7fb3fc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.509 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.509+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb4081f75f0 con 0x7fb4080b45f0 2024-04-25T14:06:50.726 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.726+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb408050740 con 0x7fb4080b45f0 2024-04-25T14:06:50.727 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.727+0000 7fb3effff640 1 -- 172.21.15.150:0/1509767909 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7fb3f805f6d0 con 0x7fb4080b45f0 2024-04-25T14:06:50.727 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:50.727 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:50.727 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 msgr2=0x7fb3f40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 0x7fb3f40769c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc034000 tx=0x7fb3fc02e000 comp rx=0 tx=0).stop 2024-04-25T14:06:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 msgr2=0x7fb4081f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800d800 tx=0x7fb3f800dcc0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.729 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 shutdown_connections 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f40744b0 0x7fb3f40769c0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb4080d1ae0 0x7fb4081eb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4080cebe0 0x7fb4081e9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 --2- 172.21.15.150:0/1509767909 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4080b45f0 0x7fb4081f65d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 >> 172.21.15.150:0/1509767909 conn(0x7fb4080b76d0 msgr2=0x7fb4080704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 shutdown_connections 2024-04-25T14:06:50.730 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.729+0000 7fb40dc05640 1 -- 172.21.15.150:0/1509767909 wait complete. 2024-04-25T14:06:50.740 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:06:50.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.849+0000 7f98db614640 1 Processor -- start 2024-04-25T14:06:50.849 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.849+0000 7f98db614640 1 -- start start 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.850+0000 7f98db614640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98d4164710 0x7f98d4160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.850+0000 7f98db614640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98d4160c00 con 0x7f98d406c960 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.850+0000 7f98db614640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f98d4160ee0 con 0x7f98d40d4840 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.850+0000 7f98db614640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f98d41611c0 con 0x7f98d4164710 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.850+0000 7f98d93b0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f98d4164710 0x7f98d4160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98d9bb1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f98d406c960 0x7f98d406cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37458/0 (socket says 172.21.15.150:37458) 2024-04-25T14:06:50.851 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98d9bb1640 1 -- 172.21.15.150:0/3380723394 learned_addr learned my addr 172.21.15.150:0/3380723394 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98d93b0640 1 -- 172.21.15.150:0/3380723394 >> v1:172.21.15.187:6789/0 conn(0x7f98d40d4840 legacy=0x7f98d40d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98d93b0640 1 -- 172.21.15.150:0/3380723394 >> v1:172.21.15.150:6789/0 conn(0x7f98d406c960 legacy=0x7f98d406cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98d93b0640 1 -- 172.21.15.150:0/3380723394 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d41614a0 con 0x7f98d4164710 2024-04-25T14:06:50.852 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.851+0000 7f98cbfff640 1 -- 172.21.15.150:0/3380723394 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130859954 0 0) 0x7f98d4160c00 con 0x7f98d406c960 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98d93b0640 1 --2- 172.21.15.150:0/3380723394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d4164710 0x7f98d4160230 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f98d00097a0 tx=0x7f98d002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab238412a4dc6195 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98cbfff640 1 -- 172.21.15.150:0/3380723394 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0037070 con 0x7f98d4164710 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98cbfff640 1 -- 172.21.15.150:0/3380723394 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98d0032440 con 0x7f98d4164710 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98cbfff640 1 -- 172.21.15.150:0/3380723394 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98d0032810 con 0x7f98d4164710 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98db614640 1 -- 172.21.15.150:0/3380723394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d4164710 msgr2=0x7f98d4160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98db614640 1 --2- 172.21.15.150:0/3380723394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d4164710 0x7f98d4160230 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f98d00097a0 tx=0x7f98d002a230 comp rx=0 tx=0).stop 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98db614640 1 -- 172.21.15.150:0/3380723394 shutdown_connections 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98db614640 1 --2- 172.21.15.150:0/3380723394 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d4164710 0x7f98d4160230 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.853 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.852+0000 7f98db614640 1 -- 172.21.15.150:0/3380723394 >> 172.21.15.150:0/3380723394 conn(0x7f98d40b77e0 msgr2=0x7f98d40b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.853+0000 7f98db614640 1 -- 172.21.15.150:0/3380723394 shutdown_connections 2024-04-25T14:06:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.853+0000 7f98db614640 1 -- 172.21.15.150:0/3380723394 wait complete. 2024-04-25T14:06:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.854+0000 7f98db614640 1 Processor -- start 2024-04-25T14:06:50.854 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.854+0000 7f98db614640 1 -- start start 2024-04-25T14:06:50.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98d406c960 0x7f98d40b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98d4164710 0x7f98d41ec170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f98d4164ca0 con 0x7f98d40d4840 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f98d4164b20 con 0x7f98d406c960 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f98d4160770 con 0x7f98d4164710 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47626/0 (socket says 172.21.15.150:47626) 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 -- 172.21.15.150:0/3962861343 learned_addr learned my addr 172.21.15.150:0/3962861343 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d9bb1640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98d4164710 0x7f98d41ec170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d93b0640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98d406c960 0x7f98d40b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 -- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98d4164710 msgr2=0x7f98d41ec170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98d4164710 0x7f98d41ec170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 -- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98d406c960 msgr2=0x7f98d40b7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:50.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98d406c960 0x7f98d40b7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 -- 172.21.15.150:0/3962861343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d40b60e0 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98d8baf640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c400 tx=0x7f98c400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4017070 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98c40052d0 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d40b63c0 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.856+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98c4003830 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.857+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d40b6980 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.858+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d4050740 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.858+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f98c401a070 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.859+0000 7f98c9ffb640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 0x7f98b80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.859+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f98c4094770 con 0x7f98d40d4840 2024-04-25T14:06:50.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.859+0000 7f98d93b0640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 0x7f98b80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:50.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.859+0000 7f98d93b0640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 0x7f98b80769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f98d0034000 tx=0x7f98d002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:50.863 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:50.863+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f98c405e690 con 0x7f98d40d4840 2024-04-25T14:06:51.038 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.038+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98d40d4110 con 0x7f98d40d4840 2024-04-25T14:06:51.039 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.040+0000 7f98c9ffb640 1 -- 172.21.15.150:0/3962861343 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+17111 (secure 0 0 0) 0x7f98c4062990 con 0x7f98d40d4840 2024-04-25T14:06:51.040 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:51.041 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":62,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:51.296371+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/133937444":"2024-04-26T14:04:37.253920+0000","172.21.15.187:6833/133937444":"2024-04-26T14:04:37.253920+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 msgr2=0x7f98b80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 0x7f98b80769c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f98d0034000 tx=0x7f98d002e000 comp rx=0 tx=0).stop 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 msgr2=0x7f98d415b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c400 tx=0x7f98c400c8c0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 shutdown_connections 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f98b80744b0 0x7f98b80769c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.042 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f98d4164710 0x7f98d41ec170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f98d40d4840 0x7f98d415b0c0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 --2- 172.21.15.150:0/3962861343 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f98d406c960 0x7f98d40b7180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 >> 172.21.15.150:0/3962861343 conn(0x7f98d40b77e0 msgr2=0x7f98d4162dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 shutdown_connections 2024-04-25T14:06:51.043 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.042+0000 7f98db614640 1 -- 172.21.15.150:0/3962861343 wait complete. 2024-04-25T14:06:51.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T14:01:56.603175+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T14:04:27.836897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T14:04:28.513769+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T14:06:51.054 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T14:06:51.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.163+0000 7fcaf47e3640 1 Processor -- start 2024-04-25T14:06:51.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.163+0000 7fcaf47e3640 1 -- start start 2024-04-25T14:06:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf47e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcaec074120 0x7fcaec0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf47e3640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcaec0ae970 0x7fcaec06f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf47e3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcaec0aed80 con 0x7fcaec147220 2024-04-25T14:06:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf47e3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec0af000 con 0x7fcaec074120 2024-04-25T14:06:51.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf47e3640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcaec0af2e0 con 0x7fcaec0ae970 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcaec074120 0x7fcaec0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcaec074120 0x7fcaec0d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47636/0 (socket says 172.21.15.150:47636) 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf257f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fcaec147220 0x7fcaec073a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37472/0 (socket says 172.21.15.150:37472) 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 -- 172.21.15.150:0/646309311 learned_addr learned my addr 172.21.15.150:0/646309311 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf2d80640 1 --2- 172.21.15.150:0/646309311 >> v2:172.21.15.187:3300/0 conn(0x7fcaec0ae970 0x7fcaec06f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 -- 172.21.15.150:0/646309311 >> v2:172.21.15.187:3300/0 conn(0x7fcaec0ae970 msgr2=0x7fcaec06f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 --2- 172.21.15.150:0/646309311 >> v2:172.21.15.187:3300/0 conn(0x7fcaec0ae970 0x7fcaec06f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 -- 172.21.15.150:0/646309311 >> v1:172.21.15.150:6789/0 conn(0x7fcaec147220 legacy=0x7fcaec073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf157d640 1 -- 172.21.15.150:0/646309311 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553075153 0 0) 0x7fcae8001490 con 0x7fcaec147220 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.165+0000 7fcaf1d7e640 1 -- 172.21.15.150:0/646309311 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec0af5c0 con 0x7fcaec074120 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.166+0000 7fcaf1d7e640 1 --2- 172.21.15.150:0/646309311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec0d0ef0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcaec071530 tx=0x7fcadc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90ee5252d906a533 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.166+0000 7fcaf157d640 1 -- 172.21.15.150:0/646309311 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc037070 con 0x7fcaec074120 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.166+0000 7fcaf157d640 1 -- 172.21.15.150:0/646309311 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcadc032440 con 0x7fcaec074120 2024-04-25T14:06:51.167 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.166+0000 7fcaf157d640 1 -- 172.21.15.150:0/646309311 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc032810 con 0x7fcaec074120 2024-04-25T14:06:51.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 -- 172.21.15.150:0/646309311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 msgr2=0x7fcaec0d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/646309311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec0d0ef0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcaec071530 tx=0x7fcadc02a270 comp rx=0 tx=0).stop 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 -- 172.21.15.150:0/646309311 shutdown_connections 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/646309311 >> v2:172.21.15.187:3300/0 conn(0x7fcaec0ae970 0x7fcaec06f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/646309311 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec0d0ef0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 -- 172.21.15.150:0/646309311 >> 172.21.15.150:0/646309311 conn(0x7fcaec0d1ee0 msgr2=0x7fcaec073300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 -- 172.21.15.150:0/646309311 shutdown_connections 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.167+0000 7fcaf47e3640 1 -- 172.21.15.150:0/646309311 wait complete. 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.169+0000 7fcaf47e3640 1 Processor -- start 2024-04-25T14:06:51.169 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.169+0000 7fcaf47e3640 1 -- start start 2024-04-25T14:06:51.171 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcaec0ae970 0x7fcaec140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec147220 0x7fcaec13b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec147a20 con 0x7fcaec074120 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcaec0af8a0 con 0x7fcaec147220 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf47e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcaec06fb80 con 0x7fcaec0ae970 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47652/0 (socket says 172.21.15.150:47652) 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 -- 172.21.15.150:0/1952038947 learned_addr learned my addr 172.21.15.150:0/1952038947 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf1d7e640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcaec0ae970 0x7fcaec140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf2d80640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec147220 0x7fcaec13b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 -- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcaec0ae970 msgr2=0x7fcaec140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcaec0ae970 0x7fcaec140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.172 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 -- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec147220 msgr2=0x7fcaec13b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec147220 0x7fcaec13b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.170+0000 7fcaf257f640 1 -- 172.21.15.150:0/1952038947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcaec1f7990 con 0x7fcaec074120 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcaf257f640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c7c0 tx=0x7fcae800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae8018070 con 0x7fcaec074120 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae800a640 con 0x7fcaec074120 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcaec1f7c70 con 0x7fcaec074120 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae800aa30 con 0x7fcaec074120 2024-04-25T14:06:51.173 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.171+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcaec1f8230 con 0x7fcaec074120 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.172+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcaec050740 con 0x7fcaec074120 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.172+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcae801b070 con 0x7fcaec074120 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.173+0000 7fcae2ffd640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.173+0000 7fcaf1d7e640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.173+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fcae8094b20 con 0x7fcaec074120 2024-04-25T14:06:51.174 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.173+0000 7fcaf1d7e640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 0x7fcad00769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcadc034000 tx=0x7fcadc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.177 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.178+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcaec0aed80 con 0x7fcaec074120 2024-04-25T14:06:51.351 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.351+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcaec070c10 con 0x7fcaec074120 2024-04-25T14:06:51.352 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.351+0000 7fcae2ffd640 1 -- 172.21.15.150:0/1952038947 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v62) ==== 93+0+10 (secure 0 0 0) 0x7fcae805f090 con 0x7fcaec074120 2024-04-25T14:06:51.352 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 1 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 msgr2=0x7fcad00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 0x7fcad00769c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcadc034000 tx=0x7fcadc02e000 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 msgr2=0x7fcaec1f4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c7c0 tx=0x7fcae800cc80 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 shutdown_connections 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcad00744b0 0x7fcad00769c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcaec147220 0x7fcaec13b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcaec0ae970 0x7fcaec140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 --2- 172.21.15.150:0/1952038947 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcaec074120 0x7fcaec1f4d00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.354 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 >> 172.21.15.150:0/1952038947 conn(0x7fcaec0d1ee0 msgr2=0x7fcaec140640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 shutdown_connections 2024-04-25T14:06:51.355 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.353+0000 7fcaf47e3640 1 -- 172.21.15.150:0/1952038947 wait complete. 2024-04-25T14:06:51.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T14:06:51.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.474+0000 7efc58a51640 1 Processor -- start 2024-04-25T14:06:51.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.474+0000 7efc58a51640 1 -- start start 2024-04-25T14:06:51.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc58a51640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc58a51640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 0x7efc540b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc58a51640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc54160810 con 0x7efc540d1ae0 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc58a51640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efc540d0300 con 0x7efc540b45f0 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc58a51640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efc54164e30 con 0x7efc540cebe0 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc51d9b640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 0x7efc540b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.187:3301/0 says I am v2:172.21.15.150:43922/0 (socket says 172.21.15.150:43922) 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 -- 172.21.15.150:0/1677812683 learned_addr learned my addr 172.21.15.150:0/1677812683 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 -- 172.21.15.150:0/1677812683 >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 msgr2=0x7efc540b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 --2- 172.21.15.150:0/1677812683 >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 0x7efc540b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 -- 172.21.15.150:0/1677812683 >> v1:172.21.15.187:6789/0 conn(0x7efc540d1ae0 legacy=0x7efc540d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc5259c640 1 -- 172.21.15.150:0/1677812683 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc540b4f40 con 0x7efc540cebe0 2024-04-25T14:06:51.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.476+0000 7efc51d9b640 1 --2- 172.21.15.150:0/1677812683 >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 0x7efc540b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.477+0000 7efc5259c640 1 --2- 172.21.15.150:0/1677812683 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efc44009880 tx=0x7efc4402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66720399f4ce89d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.477+0000 7efc5159a640 1 -- 172.21.15.150:0/1677812683 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44004030 con 0x7efc540cebe0 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.477+0000 7efc5159a640 1 -- 172.21.15.150:0/1677812683 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc440041d0 con 0x7efc540cebe0 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.477+0000 7efc5159a640 1 -- 172.21.15.150:0/1677812683 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44003610 con 0x7efc540cebe0 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 -- 172.21.15.150:0/1677812683 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 msgr2=0x7efc541648f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 --2- 172.21.15.150:0/1677812683 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7efc44009880 tx=0x7efc4402a270 comp rx=0 tx=0).stop 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 -- 172.21.15.150:0/1677812683 shutdown_connections 2024-04-25T14:06:51.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 --2- 172.21.15.150:0/1677812683 >> v2:172.21.15.187:3300/0 conn(0x7efc540b45f0 0x7efc540b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 --2- 172.21.15.150:0/1677812683 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540cebe0 0x7efc541648f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 -- 172.21.15.150:0/1677812683 >> 172.21.15.150:0/1677812683 conn(0x7efc540b76d0 msgr2=0x7efc540b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 -- 172.21.15.150:0/1677812683 shutdown_connections 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.478+0000 7efc58a51640 1 -- 172.21.15.150:0/1677812683 wait complete. 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.480+0000 7efc58a51640 1 Processor -- start 2024-04-25T14:06:51.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.480+0000 7efc58a51640 1 -- start start 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 0x7efc541609e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc540d1ae0 0x7efc5414d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efc540cfe10 con 0x7efc540cebe0 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efc540b4160 con 0x7efc540d1ae0 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc58a51640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efc540cfb30 con 0x7efc540b45f0 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc51d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.483 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc51d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47664/0 (socket says 172.21.15.150:47664) 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc5259c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 0x7efc541609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc51d9b640 1 -- 172.21.15.150:0/307308213 learned_addr learned my addr 172.21.15.150:0/307308213 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.481+0000 7efc52d9d640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc540d1ae0 0x7efc5414d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 -- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 msgr2=0x7efc541609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 0x7efc541609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 -- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc540d1ae0 msgr2=0x7efc5414d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc540d1ae0 0x7efc5414d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 -- 172.21.15.150:0/307308213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc541f6380 con 0x7efc540cebe0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc5259c640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 0x7efc541609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc51d9b640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e8e0 tx=0x7efc4800eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc48017070 con 0x7efc540cebe0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc4800c940 con 0x7efc540cebe0 2024-04-25T14:06:51.484 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc541631b0 con 0x7efc540cebe0 2024-04-25T14:06:51.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4800cd10 con 0x7efc540cebe0 2024-04-25T14:06:51.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.482+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc54163770 con 0x7efc540cebe0 2024-04-25T14:06:51.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.483+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc54050740 con 0x7efc540cebe0 2024-04-25T14:06:51.485 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.484+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efc4801a070 con 0x7efc540cebe0 2024-04-25T14:06:51.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.484+0000 7efc42ffd640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 0x7efc340769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.484+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7efc48095680 con 0x7efc540cebe0 2024-04-25T14:06:51.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.484+0000 7efc5259c640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 0x7efc340769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.486 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.485+0000 7efc5259c640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 0x7efc340769c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efc44033000 tx=0x7efc4402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.489 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.489+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efc4805f5b0 con 0x7efc540cebe0 2024-04-25T14:06:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.663+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efc540c7430 con 0x7efc540cebe0 2024-04-25T14:06:51.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.663+0000 7efc42ffd640 1 -- 172.21.15.150:0/307308213 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v62) ==== 104+0+11 (secure 0 0 0) 0x7efc480638b0 con 0x7efc540cebe0 2024-04-25T14:06:51.664 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 32 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 msgr2=0x7efc340769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 0x7efc340769c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7efc44033000 tx=0x7efc4402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 msgr2=0x7efc5414cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7efc4800e8e0 tx=0x7efc4800eda0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 shutdown_connections 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efc340744b0 0x7efc340769c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efc540d1ae0 0x7efc5414d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efc540cebe0 0x7efc5414cc30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.666 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 --2- 172.21.15.150:0/307308213 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efc540b45f0 0x7efc541609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 >> 172.21.15.150:0/307308213 conn(0x7efc540b76d0 msgr2=0x7efc54070570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 shutdown_connections 2024-04-25T14:06:51.667 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.665+0000 7efc58a51640 1 -- 172.21.15.150:0/307308213 wait complete. 2024-04-25T14:06:51.677 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T14:06:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.788+0000 7fe874451640 1 Processor -- start 2024-04-25T14:06:51.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.788+0000 7fe874451640 1 -- start start 2024-04-25T14:06:51.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.789+0000 7fe874451640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c1620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe874451640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c0b5580 0x7fe86c0b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe874451640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86c15e5e0 con 0x7fe86c15e1d0 2024-04-25T14:06:51.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe874451640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe86c162620 con 0x7fe86c0b5580 2024-04-25T14:06:51.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe874451640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0b5ed0 con 0x7fe86c161cd0 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c0b5580 0x7fe86c0b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe86c0b5580 0x7fe86c0b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47670/0 (socket says 172.21.15.150:47670) 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 -- 172.21.15.150:0/2808186990 learned_addr learned my addr 172.21.15.150:0/2808186990 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8719ec640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c1620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 -- 172.21.15.150:0/2808186990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 msgr2=0x7fe86c1620e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c1620e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 -- 172.21.15.150:0/2808186990 >> v1:172.21.15.187:6789/0 conn(0x7fe86c15e1d0 legacy=0x7fe86c0b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:51.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.790+0000 7fe8729ee640 1 -- 172.21.15.150:0/2808186990 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c162840 con 0x7fe86c0b5580 2024-04-25T14:06:51.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.791+0000 7fe8729ee640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c0b5580 0x7fe86c0b5990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe864009870 tx=0x7fe86402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fa815fd69f9706 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.791+0000 7fe8711eb640 1 -- 172.21.15.150:0/2808186990 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864037070 con 0x7fe86c0b5580 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.791+0000 7fe8711eb640 1 -- 172.21.15.150:0/2808186990 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe864032440 con 0x7fe86c0b5580 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.791+0000 7fe8711eb640 1 -- 172.21.15.150:0/2808186990 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864032810 con 0x7fe86c0b5580 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 -- 172.21.15.150:0/2808186990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c0b5580 msgr2=0x7fe86c0b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c0b5580 0x7fe86c0b5990 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fe864009870 tx=0x7fe86402a210 comp rx=0 tx=0).stop 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 -- 172.21.15.150:0/2808186990 shutdown_connections 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c0b5580 0x7fe86c0b5990 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 --2- 172.21.15.150:0/2808186990 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c1620e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 -- 172.21.15.150:0/2808186990 >> 172.21.15.150:0/2808186990 conn(0x7fe86c0d0580 msgr2=0x7fe86c0d0990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 -- 172.21.15.150:0/2808186990 shutdown_connections 2024-04-25T14:06:51.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.792+0000 7fe874451640 1 -- 172.21.15.150:0/2808186990 wait complete. 2024-04-25T14:06:51.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.793+0000 7fe874451640 1 Processor -- start 2024-04-25T14:06:51.794 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.793+0000 7fe874451640 1 -- start start 2024-04-25T14:06:51.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 0x7fe86c0cabb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.796 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c15ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0d27b0 con 0x7fe86c15e1d0 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c0d24d0 con 0x7fe86c0b5580 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe874451640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe86c0ce7a0 con 0x7fe86c161cd0 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47676/0 (socket says 172.21.15.150:47676) 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 -- 172.21.15.150:0/2353716349 learned_addr learned my addr 172.21.15.150:0/2353716349 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8729ee640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c15ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8721ed640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 0x7fe86c0cabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 -- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 msgr2=0x7fe86c15ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c15ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.797 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 -- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 msgr2=0x7fe86c0cabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 0x7fe86c0cabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 -- 172.21.15.150:0/2353716349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c0cb3a0 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8729ee640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8721ed640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 0x7fe86c0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe8719ec640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00d620 tx=0x7fe85c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00b620 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.795+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c019070 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.796+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00ba00 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.796+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c0cb680 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.796+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c0cbc40 con 0x7fe86c15e1d0 2024-04-25T14:06:51.798 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.796+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86c068c70 con 0x7fe86c15e1d0 2024-04-25T14:06:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.797+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe85c00bba0 con 0x7fe86c15e1d0 2024-04-25T14:06:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.797+0000 7fe862ffd640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 0x7fe8500769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.798+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fe85c095880 con 0x7fe86c15e1d0 2024-04-25T14:06:51.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.798+0000 7fe8721ed640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 0x7fe8500769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:51.800 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.798+0000 7fe8721ed640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 0x7fe8500769f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe868009fc0 tx=0x7fe868009550 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:51.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.802+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe86c068c70 con 0x7fe86c15e1d0 2024-04-25T14:06:51.976 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.976+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe86c0cb130 con 0x7fe86c15e1d0 2024-04-25T14:06:51.977 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.977+0000 7fe862ffd640 1 -- 172.21.15.150:0/2353716349 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v62) ==== 100+0+11 (secure 0 0 0) 0x7fe85c066e20 con 0x7fe86c15e1d0 2024-04-25T14:06:51.977 INFO:teuthology.orchestra.run.smithi150.stdout:pg_num: 64 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.978+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 msgr2=0x7fe8500769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.978+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 0x7fe8500769f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe868009fc0 tx=0x7fe868009550 comp rx=0 tx=0).stop 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.978+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 msgr2=0x7fe86c0cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.978+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fe85c00d620 tx=0x7fe85c00dae0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 shutdown_connections 2024-04-25T14:06:51.979 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe8500744e0 0x7fe8500769f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe86c161cd0 0x7fe86c15ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe86c15e1d0 0x7fe86c0cd630 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 --2- 172.21.15.150:0/2353716349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe86c0b5580 0x7fe86c0cabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 >> 172.21.15.150:0/2353716349 conn(0x7fe86c0d0580 msgr2=0x7fe86c0b7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 shutdown_connections 2024-04-25T14:06:51.980 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:51.979+0000 7fe874451640 1 -- 172.21.15.150:0/2353716349 wait complete. 2024-04-25T14:06:51.990 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:52.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.097+0000 7f01a75e3640 1 Processor -- start 2024-04-25T14:06:52.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.098+0000 7f01a75e3640 1 -- start start 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a75e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a00b9440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a75e3640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01a0160300 0x7f01a00d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a75e3640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01a00d5570 con 0x7f01a00d11d0 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a75e3640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f01a006cc10 con 0x7f01a0160300 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a75e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01a00d5850 con 0x7f01a00d44b0 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a4b7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01a0160300 0x7f01a00d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a5b80640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f01a00d11d0 0x7f01a0163ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37480/0 (socket says 172.21.15.150:37480) 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a4b7e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f01a0160300 0x7f01a00d0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47692/0 (socket says 172.21.15.150:47692) 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a5b80640 1 -- 172.21.15.150:0/3920852740 learned_addr learned my addr 172.21.15.150:0/3920852740 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a537f640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a00b9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a4b7e640 1 -- 172.21.15.150:0/3920852740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 msgr2=0x7f01a00b9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a4b7e640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a00b9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f01a4b7e640 1 -- 172.21.15.150:0/3920852740 >> v1:172.21.15.150:6789/0 conn(0x7f01a00d11d0 legacy=0x7f01a0163ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.100+0000 7f01a4b7e640 1 -- 172.21.15.150:0/3920852740 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a006c5c0 con 0x7f01a0160300 2024-04-25T14:06:52.101 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.099+0000 7f019ffff640 1 -- 172.21.15.150:0/3920852740 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3980878906 0 0) 0x7f01a00d5570 con 0x7f01a00d11d0 2024-04-25T14:06:52.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.100+0000 7f01a4b7e640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a0160300 0x7f01a00d0c90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f01940097a0 tx=0x7f019402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7509b83bd10a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.100+0000 7f019ffff640 1 -- 172.21.15.150:0/3920852740 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194037070 con 0x7f01a0160300 2024-04-25T14:06:52.102 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.100+0000 7f019ffff640 1 -- 172.21.15.150:0/3920852740 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0194032440 con 0x7f01a0160300 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.100+0000 7f019ffff640 1 -- 172.21.15.150:0/3920852740 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0194032810 con 0x7f01a0160300 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 -- 172.21.15.150:0/3920852740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a0160300 msgr2=0x7f01a00d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a0160300 0x7f01a00d0c90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f01940097a0 tx=0x7f019402a180 comp rx=0 tx=0).stop 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 -- 172.21.15.150:0/3920852740 shutdown_connections 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a0160300 0x7f01a00d0c90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 --2- 172.21.15.150:0/3920852740 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a00b9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 -- 172.21.15.150:0/3920852740 >> 172.21.15.150:0/3920852740 conn(0x7f01a00d6690 msgr2=0x7f01a00b7840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 -- 172.21.15.150:0/3920852740 shutdown_connections 2024-04-25T14:06:52.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.101+0000 7f01a75e3640 1 -- 172.21.15.150:0/3920852740 wait complete. 2024-04-25T14:06:52.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.103+0000 7f01a75e3640 1 Processor -- start 2024-04-25T14:06:52.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.103+0000 7f01a75e3640 1 -- start start 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a014dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 0x7f01a014e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f01a00d23f0 con 0x7f01a00d11d0 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f01a0160710 con 0x7f01a0160300 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a75e3640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f01a00d2570 con 0x7f01a00d44b0 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a537f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a537f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47708/0 (socket says 172.21.15.150:47708) 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.104+0000 7f01a537f640 1 -- 172.21.15.150:0/1187341585 learned_addr learned my addr 172.21.15.150:0/1187341585 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a4b7e640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a014dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a5b80640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 0x7f01a014e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 -- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 msgr2=0x7f01a014dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a014dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 -- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 msgr2=0x7f01a014e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 0x7f01a014e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 -- 172.21.15.150:0/1187341585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a014eb10 con 0x7f01a00d11d0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a5b80640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 0x7f01a014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a537f640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f019800efc0 tx=0x7f019800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198018020 con 0x7f01a00d11d0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0198014480 con 0x7f01a00d11d0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a00cb5c0 con 0x7f01a00d11d0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0198014850 con 0x7f01a00d11d0 2024-04-25T14:06:52.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.105+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a00cbb80 con 0x7f01a00d11d0 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.106+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a00cc240 con 0x7f01a00d11d0 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.107+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f0198005d10 con 0x7f01a00d11d0 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.108+0000 7f019dffb640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 0x7f018c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.108+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f0198096140 con 0x7f01a00d11d0 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.108+0000 7f01a4b7e640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 0x7f018c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.108+0000 7f01a4b7e640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 0x7f018c0769c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0194034000 tx=0x7f019402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.112 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.112+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0198022030 con 0x7f01a00d11d0 2024-04-25T14:06:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.325+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f01a0050740 con 0x7f01a00d11d0 2024-04-25T14:06:52.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.325+0000 7f019dffb640 1 -- 172.21.15.150:0/1187341585 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f01a014eb10 con 0x7f01a00d11d0 2024-04-25T14:06:52.326 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:52.326 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:52.326 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 msgr2=0x7f018c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 0x7f018c0769c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f0194034000 tx=0x7f019402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 msgr2=0x7f01a01f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f019800efc0 tx=0x7f019800c490 comp rx=0 tx=0).stop 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 shutdown_connections 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f018c0744b0 0x7f018c0769c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f01a0160300 0x7f01a014e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f01a00d44b0 0x7f01a014dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 --2- 172.21.15.150:0/1187341585 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f01a00d11d0 0x7f01a01f62a0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 >> 172.21.15.150:0/1187341585 conn(0x7f01a00d6690 msgr2=0x7f01a015f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 shutdown_connections 2024-04-25T14:06:52.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.328+0000 7f01a75e3640 1 -- 172.21.15.150:0/1187341585 wait complete. 2024-04-25T14:06:52.339 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:06:52.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.447+0000 7fee3cc96640 1 Processor -- start 2024-04-25T14:06:52.447 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.447+0000 7fee3cc96640 1 -- start start 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.448+0000 7fee3cc96640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 0x7fee380b5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.448+0000 7fee3cc96640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee3806ec50 0x7fee3806f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.448+0000 7fee3cc96640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee380b5e70 con 0x7fee380b2d10 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.448+0000 7fee3cc96640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fee38164ce0 con 0x7fee3806ec50 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.448+0000 7fee3cc96640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fee38163f20 con 0x7fee380b5520 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee3806ec50 0x7fee3806f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fee3806ec50 0x7fee3806f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47710/0 (socket says 172.21.15.150:47710) 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 -- 172.21.15.150:0/1844052723 learned_addr learned my addr 172.21.15.150:0/1844052723 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee35d9b640 1 --2- 172.21.15.150:0/1844052723 >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 0x7fee380b5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 -- 172.21.15.150:0/1844052723 >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 msgr2=0x7fee380b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 --2- 172.21.15.150:0/1844052723 >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 0x7fee380b5930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.450 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 -- 172.21.15.150:0/1844052723 >> v1:172.21.15.187:6789/0 conn(0x7fee380b2d10 legacy=0x7fee380d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee36d9d640 1 -- 172.21.15.150:0/1844052723 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee38164200 con 0x7fee3806ec50 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.449+0000 7fee35d9b640 1 --2- 172.21.15.150:0/1844052723 >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 0x7fee380b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee36d9d640 1 --2- 172.21.15.150:0/1844052723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3806ec50 0x7fee3806f060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fee380cfdf0 tx=0x7fee2802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f30d86922dcfd0e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee3559a640 1 -- 172.21.15.150:0/1844052723 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28037070 con 0x7fee3806ec50 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee3559a640 1 -- 172.21.15.150:0/1844052723 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee28032440 con 0x7fee3806ec50 2024-04-25T14:06:52.451 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee3559a640 1 -- 172.21.15.150:0/1844052723 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28032810 con 0x7fee3806ec50 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee3cc96640 1 -- 172.21.15.150:0/1844052723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3806ec50 msgr2=0x7fee3806f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.450+0000 7fee3cc96640 1 --2- 172.21.15.150:0/1844052723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3806ec50 0x7fee3806f060 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fee380cfdf0 tx=0x7fee2802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 -- 172.21.15.150:0/1844052723 shutdown_connections 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 --2- 172.21.15.150:0/1844052723 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee3806ec50 0x7fee3806f060 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 --2- 172.21.15.150:0/1844052723 >> v2:172.21.15.187:3300/0 conn(0x7fee380b5520 0x7fee380b5930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 -- 172.21.15.150:0/1844052723 >> 172.21.15.150:0/1844052723 conn(0x7fee380d0520 msgr2=0x7fee380d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 -- 172.21.15.150:0/1844052723 shutdown_connections 2024-04-25T14:06:52.452 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.451+0000 7fee3cc96640 1 -- 172.21.15.150:0/1844052723 wait complete. 2024-04-25T14:06:52.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.453+0000 7fee3cc96640 1 Processor -- start 2024-04-25T14:06:52.453 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.453+0000 7fee3cc96640 1 -- start start 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee3806ec50 0x7fee381ea4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee380b2d10 0x7fee3815e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fee381604c0 con 0x7fee380b5520 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fee381647c0 con 0x7fee380b2d10 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3cc96640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fee38160640 con 0x7fee3806ec50 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47720/0 (socket says 172.21.15.150:47720) 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 -- 172.21.15.150:0/2997527940 learned_addr learned my addr 172.21.15.150:0/2997527940 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee35d9b640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee380b2d10 0x7fee3815e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee3659c640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee3806ec50 0x7fee381ea4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 -- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee3806ec50 msgr2=0x7fee381ea4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee3806ec50 0x7fee381ea4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 -- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee380b2d10 msgr2=0x7fee3815e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee380b2d10 0x7fee3815e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.454+0000 7fee36d9d640 1 -- 172.21.15.150:0/2997527940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee38161830 con 0x7fee380b5520 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee36d9d640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fee2802a1e0 tx=0x7fee280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28037040 con 0x7fee380b5520 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee28042070 con 0x7fee380b5520 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee28030c90 con 0x7fee380b5520 2024-04-25T14:06:52.456 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee38161b10 con 0x7fee380b5520 2024-04-25T14:06:52.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.455+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee381620d0 con 0x7fee380b5520 2024-04-25T14:06:52.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.456+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee381615a0 con 0x7fee380b5520 2024-04-25T14:06:52.457 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.456+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fee28047460 con 0x7fee380b5520 2024-04-25T14:06:52.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.457+0000 7fee26ffd640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 0x7fee180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.457+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7fee280b5f60 con 0x7fee380b5520 2024-04-25T14:06:52.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.457+0000 7fee3659c640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 0x7fee180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.457+0000 7fee3659c640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 0x7fee180769c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fee30009fd0 tx=0x7fee3000d040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.462+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fee380b5e70 con 0x7fee380b5520 2024-04-25T14:06:52.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.641+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee380b5340 con 0x7fee380b5520 2024-04-25T14:06:52.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.642+0000 7fee26ffd640 1 -- 172.21.15.150:0/2997527940 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+17111 (secure 0 0 0) 0x7fee380b5340 con 0x7fee380b5520 2024-04-25T14:06:52.642 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:52.642 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":62,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:51.296371+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/133937444":"2024-04-26T14:04:37.253920+0000","172.21.15.187:6833/133937444":"2024-04-26T14:04:37.253920+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:06:52.644 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 msgr2=0x7fee180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 0x7fee180769c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fee30009fd0 tx=0x7fee3000d040 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 msgr2=0x7fee3815e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fee2802a1e0 tx=0x7fee280046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 shutdown_connections 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fee180744b0 0x7fee180769c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fee380b5520 0x7fee3815e6a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fee380b2d10 0x7fee3815e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 --2- 172.21.15.150:0/2997527940 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fee3806ec50 0x7fee381ea4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 >> 172.21.15.150:0/2997527940 conn(0x7fee380d0520 msgr2=0x7fee380b3400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 shutdown_connections 2024-04-25T14:06:52.645 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.644+0000 7fee3cc96640 1 -- 172.21.15.150:0/2997527940 wait complete. 2024-04-25T14:06:52.656 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T14:06:52.656 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:52.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.765+0000 7f6d31029640 1 Processor -- start 2024-04-25T14:06:52.765 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.765+0000 7f6d31029640 1 -- start start 2024-04-25T14:06:52.766 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d31029640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d2c164710 0x7f6d2c160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d31029640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c0d4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d31029640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d2c0d5190 con 0x7f6d2c0d09a0 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d31029640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6d2c160c00 con 0x7f6d2c164710 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d31029640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2c160ee0 con 0x7f6d2c0d4840 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d2c164710 0x7f6d2c160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6d2c164710 0x7f6d2c160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47734/0 (socket says 172.21.15.150:47734) 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 -- 172.21.15.150:0/245914102 learned_addr learned my addr 172.21.15.150:0/245914102 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2a59c640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c0d4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 -- 172.21.15.150:0/245914102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 msgr2=0x7f6d2c0d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c0d4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.767 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 -- 172.21.15.150:0/245914102 >> v1:172.21.15.187:6789/0 conn(0x7f6d2c0d09a0 legacy=0x7f6d2c0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:52.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2ad9d640 1 -- 172.21.15.150:0/245914102 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c1611c0 con 0x7f6d2c164710 2024-04-25T14:06:52.768 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.766+0000 7f6d2a59c640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c0d4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.767+0000 7f6d2ad9d640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c160230 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009e50 tx=0x7f6d1c02a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e4786721a7a37c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.767+0000 7f6d29d9b640 1 -- 172.21.15.150:0/245914102 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c036070 con 0x7f6d2c164710 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.767+0000 7f6d29d9b640 1 -- 172.21.15.150:0/245914102 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1c031600 con 0x7f6d2c164710 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.767+0000 7f6d29d9b640 1 -- 172.21.15.150:0/245914102 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c0319d0 con 0x7f6d2c164710 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 -- 172.21.15.150:0/245914102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 msgr2=0x7f6d2c160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c160230 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009e50 tx=0x7f6d1c02a6c0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 -- 172.21.15.150:0/245914102 shutdown_connections 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c0d4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 --2- 172.21.15.150:0/245914102 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c160230 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.769 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 -- 172.21.15.150:0/245914102 >> 172.21.15.150:0/245914102 conn(0x7f6d2c0b77e0 msgr2=0x7f6d2c0b7bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 -- 172.21.15.150:0/245914102 shutdown_connections 2024-04-25T14:06:52.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.768+0000 7f6d31029640 1 -- 172.21.15.150:0/245914102 wait complete. 2024-04-25T14:06:52.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.770+0000 7f6d31029640 1 Processor -- start 2024-04-25T14:06:52.770 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.770+0000 7f6d31029640 1 -- start start 2024-04-25T14:06:52.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d2c0d09a0 0x7f6d2c15e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c1f9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.773 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c164b20 con 0x7f6d2c164710 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2c0d5630 con 0x7f6d2c0d09a0 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d31029640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2c164ca0 con 0x7f6d2c0d4840 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d2b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d2b59e640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47744/0 (socket says 172.21.15.150:47744) 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d2b59e640 1 -- 172.21.15.150:0/3737438313 learned_addr learned my addr 172.21.15.150:0/3737438313 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d2a59c640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c1f9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.771+0000 7f6d2ad9d640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d2c0d09a0 0x7f6d2c15e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 -- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 msgr2=0x7f6d2c1f9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c1f9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.774 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 -- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d2c0d09a0 msgr2=0x7f6d2c15e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d2c0d09a0 0x7f6d2c15e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 -- 172.21.15.150:0/3737438313 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2c1fa3b0 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d2b59e640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b3c0 tx=0x7f6d2400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24011830 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d24016070 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d24011c30 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2c1fa690 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.772+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2c1ec4d0 con 0x7f6d2c164710 2024-04-25T14:06:52.775 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.773+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2c0ca320 con 0x7f6d2c164710 2024-04-25T14:06:52.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.774+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6d2401b460 con 0x7f6d2c164710 2024-04-25T14:06:52.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.774+0000 7f6d137fe640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 0x7f6d180769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:52.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.774+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f6d24094e60 con 0x7f6d2c164710 2024-04-25T14:06:52.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.774+0000 7f6d2ad9d640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 0x7f6d180769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:52.776 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.775+0000 7f6d2ad9d640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 0x7f6d180769c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c033000 tx=0x7f6d1c009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:52.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.779+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d2c0ca320 con 0x7f6d2c164710 2024-04-25T14:06:52.989 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.989+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6d2c050740 con 0x7f6d2c164710 2024-04-25T14:06:52.990 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.990+0000 7f6d137fe640 1 -- 172.21.15.150:0/3737438313 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f6d2405ee10 con 0x7f6d2c164710 2024-04-25T14:06:52.990 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:52.990 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:52.990 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:52.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 msgr2=0x7f6d180769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.992 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 0x7f6d180769c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c033000 tx=0x7f6d1c009000 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 msgr2=0x7f6d2c0cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f6d2400b3c0 tx=0x7f6d2400b880 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 shutdown_connections 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6d180744b0 0x7f6d180769c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6d2c164710 0x7f6d2c0cc9e0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6d2c0d4840 0x7f6d2c1f9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 --2- 172.21.15.150:0/3737438313 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6d2c0d09a0 0x7f6d2c15e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 >> 172.21.15.150:0/3737438313 conn(0x7f6d2c0b77e0 msgr2=0x7f6d2c068760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 shutdown_connections 2024-04-25T14:06:52.993 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:52.992+0000 7f6d31029640 1 -- 172.21.15.150:0/3737438313 wait complete. 2024-04-25T14:06:53.004 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:06:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.111+0000 7f15c2cd9640 1 Processor -- start 2024-04-25T14:06:53.111 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.111+0000 7f15c2cd9640 1 -- start start 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c2cd9640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f15bc0cebe0 0x7f15bc1648f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c2cd9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc0b4a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c2cd9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15bc160810 con 0x7f15bc0d1ae0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c2cd9640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f15bc0d0300 con 0x7f15bc0cebe0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c2cd9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15bc164e30 con 0x7f15bc0b45f0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c1276640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f15bc0d1ae0 0x7f15bc0d1ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37494/0 (socket says 172.21.15.150:37494) 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c1276640 1 -- 172.21.15.150:0/3204679144 learned_addr learned my addr 172.21.15.150:0/3204679144 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3204679144 >> v2:172.21.15.187:3300/0 conn(0x7f15bc0cebe0 0x7f15bc1648f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15bbfff640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc0b4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1054167941 0 0) 0x7f15bc160810 con 0x7f15bc0d1ae0 2024-04-25T14:06:53.114 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15a4003520 con 0x7f15bc0d1ae0 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c0a75640 1 -- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 msgr2=0x7f15bc0b4a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc0b4a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2311129318 0 0) 0x7f15a4003520 con 0x7f15bc0d1ae0 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c0a75640 1 -- 172.21.15.150:0/3204679144 >> v1:172.21.15.150:6789/0 conn(0x7f15bc0d1ae0 legacy=0x7f15bc0d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15c0a75640 1 -- 172.21.15.150:0/3204679144 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc0b4f40 con 0x7f15bc0cebe0 2024-04-25T14:06:53.115 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.113+0000 7f15bbfff640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:53.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.114+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0cebe0 0x7f15bc1648f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0097c0 tx=0x7f15ac02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c17ddb31d58f07c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.114+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac037070 con 0x7f15bc0cebe0 2024-04-25T14:06:53.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.114+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15ac02abc0 con 0x7f15bc0cebe0 2024-04-25T14:06:53.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.114+0000 7f15bb7fe640 1 -- 172.21.15.150:0/3204679144 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b40019f0 con 0x7f15bc0cebe0 2024-04-25T14:06:53.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0cebe0 msgr2=0x7f15bc1648f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0cebe0 0x7f15bc1648f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0097c0 tx=0x7f15ac02a130 comp rx=0 tx=0).stop 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3204679144 shutdown_connections 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc0b4a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3204679144 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0cebe0 0x7f15bc1648f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3204679144 >> 172.21.15.150:0/3204679144 conn(0x7f15bc0b76d0 msgr2=0x7f15bc0b7ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3204679144 shutdown_connections 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.115+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3204679144 wait complete. 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.117+0000 7f15c2cd9640 1 Processor -- start 2024-04-25T14:06:53.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.117+0000 7f15c2cd9640 1 -- start start 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc1ea980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0d1ae0 0x7f15bc0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc0cfe10 con 0x7f15bc0cebe0 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc0b4160 con 0x7f15bc0d1ae0 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15c2cd9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f15bc0cfb30 con 0x7f15bc0b45f0 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15bbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47758/0 (socket says 172.21.15.150:47758) 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.118+0000 7f15bbfff640 1 -- 172.21.15.150:0/3961218338 learned_addr learned my addr 172.21.15.150:0/3961218338 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc1ea980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15c1276640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0d1ae0 0x7f15bc0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 -- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 msgr2=0x7f15bc1ea980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc1ea980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 -- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0d1ae0 msgr2=0x7f15bc0c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0d1ae0 0x7f15bc0c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 -- 172.21.15.150:0/3961218338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc14c340 con 0x7f15bc0cebe0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc1ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15bbfff640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f15b000e960 tx=0x7f15b000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0017070 con 0x7f15bc0cebe0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15b000ca00 con 0x7f15bc0cebe0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b000cdd0 con 0x7f15bc0cebe0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc0c6b80 con 0x7f15bc0cebe0 2024-04-25T14:06:53.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.119+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc0c7140 con 0x7f15bc0cebe0 2024-04-25T14:06:53.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.120+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15bc050740 con 0x7f15bc0cebe0 2024-04-25T14:06:53.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.121+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f15b001a070 con 0x7f15bc0cebe0 2024-04-25T14:06:53.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.121+0000 7f15b97fa640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 0x7f15a80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.121+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 0x7f15a80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.121+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f15b0095230 con 0x7f15bc0cebe0 2024-04-25T14:06:53.123 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.122+0000 7f15c0a75640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 0x7f15a80769c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f15ac034000 tx=0x7f15ac02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.126 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.126+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f15b005f1e0 con 0x7f15bc0cebe0 2024-04-25T14:06:53.341 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.341+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15bc0c6910 con 0x7f15bc0cebe0 2024-04-25T14:06:53.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.342+0000 7f15b97fa640 1 -- 172.21.15.150:0/3961218338 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) ==== 94+0+5188 (secure 0 0 0) 0x7f15b00634e0 con 0x7f15bc0cebe0 2024-04-25T14:06:53.342 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:53.342 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":37,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-25T14:04:29.564574+0000","modified":"2024-04-25T14:04:41.277594+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":33,"state":"up:active","state_seq":5,"addr":"172.21.15.150:6836/685573768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":685573768},{"type":"v1","addr":"172.21.15.150:6836","nonce":685573768}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5982,"qdb_cluster":[5982]},"id":3}]} 2024-04-25T14:06:53.343 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 37 2024-04-25T14:06:53.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 msgr2=0x7f15a80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 0x7f15a80769c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f15ac034000 tx=0x7f15ac02e000 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 msgr2=0x7f15bc14be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f15b000e960 tx=0x7f15b000ee20 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 shutdown_connections 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f15a80744b0 0x7f15a80769c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f15bc0d1ae0 0x7f15bc0c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f15bc0cebe0 0x7f15bc14be00 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 --2- 172.21.15.150:0/3961218338 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f15bc0b45f0 0x7f15bc1ea980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 >> 172.21.15.150:0/3961218338 conn(0x7f15bc0b76d0 msgr2=0x7f15bc071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 shutdown_connections 2024-04-25T14:06:53.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.344+0000 7f15c2cd9640 1 -- 172.21.15.150:0/3961218338 wait complete. 2024-04-25T14:06:53.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:06:53.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.464+0000 7f772fad8640 1 Processor -- start 2024-04-25T14:06:53.464 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.464+0000 7f772fad8640 1 -- start start 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772fad8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f77280b3160 0x7f772815e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772fad8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7728162240 con 0x7f7728071c70 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772fad8640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7728162520 con 0x7f77280ce510 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772fad8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7728162800 con 0x7f77280b3160 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772e075640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7728071c70 0x7f77280d25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37510/0 (socket says 172.21.15.150:37510) 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772e075640 1 -- 172.21.15.150:0/800558658 learned_addr learned my addr 172.21.15.150:0/800558658 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.465+0000 7f772d874640 1 --2- 172.21.15.150:0/800558658 >> v2:172.21.15.187:3300/0 conn(0x7f77280b3160 0x7f772815e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 639996885 0 0) 0x7f7728162240 con 0x7f7728071c70 2024-04-25T14:06:53.466 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7714003520 con 0x7f7728071c70 2024-04-25T14:06:53.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772d874640 1 -- 172.21.15.150:0/800558658 >> v1:172.21.15.187:6789/0 conn(0x7f77280ce510 legacy=0x7f7728071560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2429926104 0 0) 0x7f7714003520 con 0x7f7728071c70 2024-04-25T14:06:53.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772d874640 1 -- 172.21.15.150:0/800558658 >> v1:172.21.15.150:6789/0 conn(0x7f7728071c70 legacy=0x7f77280d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.467 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.466+0000 7f772d874640 1 -- 172.21.15.150:0/800558658 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7728162ae0 con 0x7f77280b3160 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.467+0000 7f772d874640 1 --2- 172.21.15.150:0/800558658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280b3160 0x7f772815e0b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77240097a0 tx=0x7f772402a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c9aab758a74681c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.467+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724037070 con 0x7f77280b3160 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.467+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f772402ab60 con 0x7f77280b3160 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.467+0000 7f772c872640 1 -- 172.21.15.150:0/800558658 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77200019f0 con 0x7f77280b3160 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 -- 172.21.15.150:0/800558658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280b3160 msgr2=0x7f772815e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 --2- 172.21.15.150:0/800558658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280b3160 0x7f772815e0b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77240097a0 tx=0x7f772402a0d0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 -- 172.21.15.150:0/800558658 shutdown_connections 2024-04-25T14:06:53.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 --2- 172.21.15.150:0/800558658 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280b3160 0x7f772815e0b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 -- 172.21.15.150:0/800558658 >> 172.21.15.150:0/800558658 conn(0x7f77280d1170 msgr2=0x7f77280d3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:53.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 -- 172.21.15.150:0/800558658 shutdown_connections 2024-04-25T14:06:53.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.468+0000 7f772fad8640 1 -- 172.21.15.150:0/800558658 wait complete. 2024-04-25T14:06:53.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.469+0000 7f772fad8640 1 Processor -- start 2024-04-25T14:06:53.469 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.469+0000 7f772fad8640 1 -- start start 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280ce510 0x7f7728158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f772815a450 0x7f772815a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7728161f50 con 0x7f77280c7ed0 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7728161dd0 con 0x7f77280ce510 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f77280d2cd0 con 0x7f772815a450 2024-04-25T14:06:53.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47774/0 (socket says 172.21.15.150:47774) 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 -- 172.21.15.150:0/2010231109 learned_addr learned my addr 172.21.15.150:0/2010231109 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772d073640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f772815a450 0x7f772815a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772d874640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280ce510 0x7f7728158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 -- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f772815a450 msgr2=0x7f772815a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f772815a450 0x7f772815a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 -- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280ce510 msgr2=0x7f7728158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280ce510 0x7f7728158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 -- 172.21.15.150:0/2010231109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77280c8820 con 0x7f77280c7ed0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772e075640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f772000c7f0 tx=0x7f772000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7720018070 con 0x7f77280c7ed0 2024-04-25T14:06:53.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f772000a640 con 0x7f77280c7ed0 2024-04-25T14:06:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f772815e800 con 0x7f77280c7ed0 2024-04-25T14:06:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.471+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772000aa30 con 0x7f77280c7ed0 2024-04-25T14:06:53.474 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.472+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f772815b140 con 0x7f77280c7ed0 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.472+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f772815e9d0 con 0x7f77280c7ed0 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.473+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f772001b070 con 0x7f77280c7ed0 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.473+0000 7f771e7fc640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 0x7f770c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.474+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f7720094980 con 0x7f77280c7ed0 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.474+0000 7f772d874640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 0x7f770c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.475 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.474+0000 7f772d874640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 0x7f770c0769c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7724034000 tx=0x7f772402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.478+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f77280c8820 con 0x7f77280c7ed0 2024-04-25T14:06:53.653 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.653+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f772815aed0 con 0x7f77280c7ed0 2024-04-25T14:06:53.654 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.654+0000 7f771e7fc640 1 -- 172.21.15.150:0/2010231109 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) ==== 74+0+17111 (secure 0 0 0) 0x7f772005e930 con 0x7f77280c7ed0 2024-04-25T14:06:53.655 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:53.655 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":62,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:04:51.296371+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-04-25T14:04:27.836897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-04-25T14:04:28.513769+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1d","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.187:6835/133937444":"2024-04-26T14:04:37.253920+0000","172.21.15.187:6833/133937444":"2024-04-26T14:04:37.253920+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 msgr2=0x7f770c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 0x7f770c0769c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7724034000 tx=0x7f772402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 msgr2=0x7f77280c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f772000c7f0 tx=0x7f772000ccb0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 shutdown_connections 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f770c0744b0 0x7f770c0769c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f77280c7ed0 0x7f77280c82e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f772815a450 0x7f772815a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 --2- 172.21.15.150:0/2010231109 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f77280ce510 0x7f7728158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 >> 172.21.15.150:0/2010231109 conn(0x7f77280d1170 msgr2=0x7f7728158910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:53.657 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 shutdown_connections 2024-04-25T14:06:53.658 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.656+0000 7f772fad8640 1 -- 172.21.15.150:0/2010231109 wait complete. 2024-04-25T14:06:53.668 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T14:06:53.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.779+0000 7f43edf26640 1 Processor -- start 2024-04-25T14:06:53.779 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.779+0000 7f43edf26640 1 -- start start 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43edf26640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f43e8169100 0x7f43e816b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43edf26640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43e8164ce0 con 0x7f43e80b5520 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43edf26640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43e8163f20 con 0x7f43e80b2d10 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43edf26640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f43e8164200 con 0x7f43e8169100 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43e6ffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f43e80b5520 0x7f43e80b5930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37514/0 (socket says 172.21.15.150:37514) 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43e6ffd640 1 -- 172.21.15.150:0/2765113110 learned_addr learned my addr 172.21.15.150:0/2765113110 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43e7fff640 1 --2- 172.21.15.150:0/2765113110 >> v2:172.21.15.187:3300/0 conn(0x7f43e8169100 0x7f43e816b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906003571 0 0) 0x7f43e8164ce0 con 0x7f43e80b5520 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.780+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43d0003520 con 0x7f43e80b5520 2024-04-25T14:06:53.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.1 v1:172.21.15.187:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258305024 0 0) 0x7f43e8163f20 con 0x7f43e80b2d10 2024-04-25T14:06:53.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 --> v1:172.21.15.187:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43e8164ce0 con 0x7f43e80b2d10 2024-04-25T14:06:53.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e7fff640 1 -- 172.21.15.150:0/2765113110 >> v1:172.21.15.187:6789/0 conn(0x7f43e80b2d10 legacy=0x7f43e80d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 792065149 0 0) 0x7f43d0003520 con 0x7f43e80b5520 2024-04-25T14:06:53.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e7fff640 1 -- 172.21.15.150:0/2765113110 >> v1:172.21.15.150:6789/0 conn(0x7f43e80b5520 legacy=0x7f43e80b5930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.782 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e7fff640 1 -- 172.21.15.150:0/2765113110 --> v2:172.21.15.187:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e81644e0 con 0x7f43e8169100 2024-04-25T14:06:53.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.781+0000 7f43e7fff640 1 --2- 172.21.15.150:0/2765113110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e816b520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eca673341514ade9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.1 v2:172.21.15.187:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8001670 con 0x7f43e8169100 2024-04-25T14:06:53.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.1 v2:172.21.15.187:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e0007540 con 0x7f43e8169100 2024-04-25T14:06:53.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43e67fc640 1 -- 172.21.15.150:0/2765113110 <== mon.1 v2:172.21.15.187:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0007910 con 0x7f43e8169100 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 -- 172.21.15.150:0/2765113110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 msgr2=0x7f43e816b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 --2- 172.21.15.150:0/2765113110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e816b520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002a5f0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 -- 172.21.15.150:0/2765113110 shutdown_connections 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 --2- 172.21.15.150:0/2765113110 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e816b520 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 -- 172.21.15.150:0/2765113110 >> 172.21.15.150:0/2765113110 conn(0x7f43e80d0520 msgr2=0x7f43e80d0930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 -- 172.21.15.150:0/2765113110 shutdown_connections 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.782+0000 7f43edf26640 1 -- 172.21.15.150:0/2765113110 wait complete. 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.784+0000 7f43edf26640 1 Processor -- start 2024-04-25T14:06:53.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.784+0000 7f43edf26640 1 -- start start 2024-04-25T14:06:53.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43e80b2d10 0x7f43e815f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e80b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f43e81604c0 con 0x7f43e80b5520 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f43e806fc60 con 0x7f43e8169100 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43edf26640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f43e8160640 con 0x7f43e80b2d10 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47776/0 (socket says 172.21.15.150:47776) 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 -- 172.21.15.150:0/258652352 learned_addr learned my addr 172.21.15.150:0/258652352 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e77fe640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43e80b2d10 0x7f43e815f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e7fff640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e80b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 -- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43e80b2d10 msgr2=0x7f43e815f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43e80b2d10 0x7f43e815f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 -- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 msgr2=0x7f43e80b7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e80b7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 -- 172.21.15.150:0/258652352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e80b6840 con 0x7f43e80b5520 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43e6ffd640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005880 tx=0x7f43d800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.788 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.786+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8017070 con 0x7f43e80b5520 2024-04-25T14:06:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.787+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d8005ad0 con 0x7f43e80b5520 2024-04-25T14:06:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.787+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e80b6b20 con 0x7f43e80b5520 2024-04-25T14:06:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.787+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8007070 con 0x7f43e80b5520 2024-04-25T14:06:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.787+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e80b70e0 con 0x7f43e80b5520 2024-04-25T14:06:53.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.788+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e8050740 con 0x7f43e80b5520 2024-04-25T14:06:53.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.788+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f43d801a070 con 0x7f43e80b5520 2024-04-25T14:06:53.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.789+0000 7f43cffff640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 0x7f43d40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:53.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.789+0000 7f43e77fe640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 0x7f43d40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:53.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.789+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(62..62 src has 1..62) ==== 6245+0+0 (secure 0 0 0) 0x7f43d80947f0 con 0x7f43e80b5520 2024-04-25T14:06:53.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.789+0000 7f43e77fe640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 0x7f43d40769c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0011e0 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:53.793 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:53.794+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f43d805e7a0 con 0x7f43e80b5520 2024-04-25T14:06:54.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.005+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f43e8163a00 con 0x7f43e80b5520 2024-04-25T14:06:54.006 INFO:tasks.ceph.mon.a.smithi150.stderr:2024-04-25T14:06:54.006+0000 7f46d47d6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T14:06:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.008+0000 7f43cffff640 1 -- 172.21.15.150:0/258652352 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v38) ==== 161+0+0 (secure 0 0 0) 0x7f43d8062aa0 con 0x7f43e80b5520 2024-04-25T14:06:54.008 INFO:teuthology.orchestra.run.smithi150.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T14:06:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 msgr2=0x7f43d40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 0x7f43d40769c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43dc0011e0 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2024-04-25T14:06:54.011 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 msgr2=0x7f43e8162670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f43d8005880 tx=0x7f43d800cbe0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 shutdown_connections 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f43d40744b0 0x7f43d40769c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f43e8169100 0x7f43e80b7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f43e80b5520 0x7f43e8162670 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 --2- 172.21.15.150:0/258652352 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f43e80b2d10 0x7f43e815f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 >> 172.21.15.150:0/258652352 conn(0x7f43e80d0520 msgr2=0x7f43e8159a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 shutdown_connections 2024-04-25T14:06:54.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.011+0000 7f43edf26640 1 -- 172.21.15.150:0/258652352 wait complete. 2024-04-25T14:06:54.014 INFO:tasks.ceph.mds.a.smithi150.stderr: -6328> 2024-04-25T14:04:42.249+0000 7f5c27a1b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T14:06:54.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T14:06:54.069 INFO:tasks.ceph.mds.a.smithi150.stdout:starting mds.a at 2024-04-25T14:06:54.134 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.134+0000 7fec20292640 1 Processor -- start 2024-04-25T14:06:54.135 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.134+0000 7fec20292640 1 -- start start 2024-04-25T14:06:54.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec20292640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec180b2c20 0x7fec18146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:54.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec20292640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec180b1170 con 0x7fec180b0bb0 2024-04-25T14:06:54.139 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec20292640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1806e430 con 0x7fec1806dee0 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec20292640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fec18146590 con 0x7fec180b2c20 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec1e82f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fec180b0bb0 0x7fec1806c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37524/0 (socket says 172.21.15.150:37524) 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec1d82d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fec180b2c20 0x7fec18146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.136+0000 7fec1e82f640 1 -- 172.21.15.150:0/4176815321 learned_addr learned my addr 172.21.15.150:0/4176815321 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d82d640 1 -- 172.21.15.150:0/4176815321 >> v1:172.21.15.187:6789/0 conn(0x7fec1806dee0 legacy=0x7fec180b04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.140 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d02c640 1 -- 172.21.15.150:0/4176815321 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784118741 0 0) 0x7fec180b1170 con 0x7fec180b0bb0 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d82d640 1 -- 172.21.15.150:0/4176815321 >> v1:172.21.15.150:6789/0 conn(0x7fec180b0bb0 legacy=0x7fec1806c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d82d640 1 -- 172.21.15.150:0/4176815321 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec180b1410 con 0x7fec180b2c20 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d82d640 1 --2- 172.21.15.150:0/4176815321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b2c20 0x7fec18146050 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec140097a0 tx=0x7fec1402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ff346793964fb3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d02c640 1 -- 172.21.15.150:0/4176815321 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14037070 con 0x7fec180b2c20 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d02c640 1 -- 172.21.15.150:0/4176815321 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec14032440 con 0x7fec180b2c20 2024-04-25T14:06:54.141 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.137+0000 7fec1d02c640 1 -- 172.21.15.150:0/4176815321 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14032810 con 0x7fec180b2c20 2024-04-25T14:06:54.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 -- 172.21.15.150:0/4176815321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b2c20 msgr2=0x7fec18146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 --2- 172.21.15.150:0/4176815321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b2c20 0x7fec18146050 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fec140097a0 tx=0x7fec1402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 -- 172.21.15.150:0/4176815321 shutdown_connections 2024-04-25T14:06:54.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 --2- 172.21.15.150:0/4176815321 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b2c20 0x7fec18146050 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.142 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 -- 172.21.15.150:0/4176815321 >> 172.21.15.150:0/4176815321 conn(0x7fec18073dc0 msgr2=0x7fec180b7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:54.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 -- 172.21.15.150:0/4176815321 shutdown_connections 2024-04-25T14:06:54.143 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.138+0000 7fec20292640 1 -- 172.21.15.150:0/4176815321 wait complete. 2024-04-25T14:06:54.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.140+0000 7fec20292640 1 Processor -- start 2024-04-25T14:06:54.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.140+0000 7fec20292640 1 -- start start 2024-04-25T14:06:54.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec1806dee0 0x7fec18138de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec180b2c20 0x7fec1813a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fec1809bda0 con 0x7fec180b0bb0 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fec1809bc20 con 0x7fec1806dee0 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec20292640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fec1806ccd0 con 0x7fec180b2c20 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec1d82d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec1d82d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47808/0 (socket says 172.21.15.150:47808) 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec1d82d640 1 -- 172.21.15.150:0/228793275 learned_addr learned my addr 172.21.15.150:0/228793275 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec1e02e640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec1806dee0 0x7fec18138de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.141+0000 7fec1e82f640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec180b2c20 0x7fec1813a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 -- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec180b2c20 msgr2=0x7fec1813a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec180b2c20 0x7fec1813a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.149 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 -- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec1806dee0 msgr2=0x7fec18138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec1806dee0 0x7fec18138de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 -- 172.21.15.150:0/228793275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1809e2e0 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec1d82d640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fec140097a0 tx=0x7fec140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14037050 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec14038040 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec1809e5c0 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec14004490 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.142+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec1815c4f0 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.143+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec18050740 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.144+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fec14030900 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.144+0000 7fec0effd640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 0x7febfc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.144+0000 7fec1e02e640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 0x7febfc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.144+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6331+0+0 (secure 0 0 0) 0x7fec140b5f40 con 0x7fec180b0bb0 2024-04-25T14:06:54.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.145+0000 7fec1e02e640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 0x7febfc0769c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fec18072da0 tx=0x7fec08009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:54.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.149+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fec1407fea0 con 0x7fec180b0bb0 2024-04-25T14:06:54.366 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:54.366+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fec1813f210 con 0x7fec180b0bb0 2024-04-25T14:06:55.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.015+0000 7fec0effd640 1 -- 172.21.15.150:0/228793275 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v39) ==== 104+0+0 (secure 0 0 0) 0x7fec140841a0 con 0x7fec180b0bb0 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 msgr2=0x7febfc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 0x7febfc0769c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fec18072da0 tx=0x7fec08009040 comp rx=0 tx=0).stop 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 msgr2=0x7fec1815bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fec140097a0 tx=0x7fec140046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 shutdown_connections 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7febfc0744b0 0x7febfc0769c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fec180b2c20 0x7fec1813a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.017+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fec180b0bb0 0x7fec1815bfb0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.018+0000 7fec20292640 1 --2- 172.21.15.150:0/228793275 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fec1806dee0 0x7fec18138de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.018+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 >> 172.21.15.150:0/228793275 conn(0x7fec18073dc0 msgr2=0x7fec1813f730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.018+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 shutdown_connections 2024-04-25T14:06:55.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.018+0000 7fec20292640 1 -- 172.21.15.150:0/228793275 wait complete. 2024-04-25T14:06:55.031 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.148+0000 7fe5a8572640 1 Processor -- start 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.148+0000 7fe5a8572640 1 -- start start 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a8572640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe5a00d1ab0 0x7fe5a0160280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a8572640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe5a00d2710 0x7fe5a006efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a8572640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5a00ceda0 con 0x7fe5a00ce530 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a8572640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fe5a00cf080 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a8572640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe5a00cf360 con 0x7fe5a00d2710 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe5a00d1ab0 0x7fe5a0160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fe5a00d1ab0 0x7fe5a0160280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47824/0 (socket says 172.21.15.150:47824) 2024-04-25T14:06:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 -- 172.21.15.150:0/908682038 learned_addr learned my addr 172.21.15.150:0/908682038 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a6b0f640 1 --2- 172.21.15.150:0/908682038 >> v2:172.21.15.187:3300/0 conn(0x7fe5a00d2710 0x7fe5a006efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 -- 172.21.15.150:0/908682038 >> v2:172.21.15.187:3300/0 conn(0x7fe5a00d2710 msgr2=0x7fe5a006efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 --2- 172.21.15.150:0/908682038 >> v2:172.21.15.187:3300/0 conn(0x7fe5a00d2710 0x7fe5a006efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 -- 172.21.15.150:0/908682038 >> v1:172.21.15.187:6789/0 conn(0x7fe5a00ce530 legacy=0x7fe5a00d2000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:55.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.150+0000 7fe5a630e640 1 -- 172.21.15.150:0/908682038 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a00cf640 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.151+0000 7fe5a630e640 1 --2- 172.21.15.150:0/908682038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a0160280 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00d4750 tx=0x7fe59c02a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bbec2e702ebe06 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:55.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.151+0000 7fe5a530c640 1 -- 172.21.15.150:0/908682038 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c036070 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.151+0000 7fe5a530c640 1 -- 172.21.15.150:0/908682038 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe59c031600 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.155 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.151+0000 7fe5a530c640 1 -- 172.21.15.150:0/908682038 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe59c0319d0 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 -- 172.21.15.150:0/908682038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00d1ab0 msgr2=0x7fe5a0160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 --2- 172.21.15.150:0/908682038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a0160280 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00d4750 tx=0x7fe59c02a590 comp rx=0 tx=0).stop 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 -- 172.21.15.150:0/908682038 shutdown_connections 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 --2- 172.21.15.150:0/908682038 >> v2:172.21.15.187:3300/0 conn(0x7fe5a00d2710 0x7fe5a006efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 --2- 172.21.15.150:0/908682038 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a0160280 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 -- 172.21.15.150:0/908682038 >> 172.21.15.150:0/908682038 conn(0x7fe5a00b5570 msgr2=0x7fe5a00b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 -- 172.21.15.150:0/908682038 shutdown_connections 2024-04-25T14:06:55.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.152+0000 7fe5a8572640 1 -- 172.21.15.150:0/908682038 wait complete. 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.153+0000 7fe5a8572640 1 Processor -- start 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.154+0000 7fe5a8572640 1 -- start start 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5a00ce530 0x7fe5a015abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a00af540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a00ceac0 con 0x7fe5a00afa80 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe5a00ce940 con 0x7fe5a00d1ab0 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a8572640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe5a006f500 con 0x7fe5a00ce530 2024-04-25T14:06:55.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47830/0 (socket says 172.21.15.150:47830) 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 -- 172.21.15.150:0/2878109349 learned_addr learned my addr 172.21.15.150:0/2878109349 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a5b0d640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a00af540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a630e640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5a00ce530 0x7fe5a015abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 -- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5a00ce530 msgr2=0x7fe5a015abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5a00ce530 0x7fe5a015abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 -- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5a00d1ab0 msgr2=0x7fe5a00af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a00af540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 -- 172.21.15.150:0/2878109349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5a00b03d0 con 0x7fe5a00afa80 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.155+0000 7fe5a6b0f640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d600 tx=0x7fe59800dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.156+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5980125f0 con 0x7fe5a00afa80 2024-04-25T14:06:55.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.156+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe598019070 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.156+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5980129f0 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.156+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5a01e9210 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.156+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5a01e5d80 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.157+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5a0050740 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.157+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe598012b90 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.158+0000 7fe596ffd640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 0x7fe5840769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.158+0000 7fe5a630e640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 0x7fe5840769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.158+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 6331+0+0 (secure 0 0 0) 0x7fe598095650 con 0x7fe5a00afa80 2024-04-25T14:06:55.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.158+0000 7fe5a630e640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 0x7fe5840769c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe59c033000 tx=0x7fe59c009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:55.163 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.163+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe59805f5c0 con 0x7fe5a00afa80 2024-04-25T14:06:55.338 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:55.337+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe5a015fac0 con 0x7fe5a00afa80 2024-04-25T14:06:56.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.018+0000 7fe596ffd640 1 -- 172.21.15.150:0/2878109349 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v64) ==== 181+0+0 (secure 0 0 0) 0x7fe5980638c0 con 0x7fe5a00afa80 2024-04-25T14:06:56.018 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_metadata' removed 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 msgr2=0x7fe5840769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 0x7fe5840769c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fe59c033000 tx=0x7fe59c009000 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 msgr2=0x7fe5a00afe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fe59800d600 tx=0x7fe59800dac0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 shutdown_connections 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe5840744b0 0x7fe5840769c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe5a00afa80 0x7fe5a00afe90 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe5a00d1ab0 0x7fe5a00af540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 --2- 172.21.15.150:0/2878109349 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe5a00ce530 0x7fe5a015abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 >> 172.21.15.150:0/2878109349 conn(0x7fe5a00b5570 msgr2=0x7fe5a00b46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.020+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 shutdown_connections 2024-04-25T14:06:56.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.021+0000 7fe5a8572640 1 -- 172.21.15.150:0/2878109349 wait complete. 2024-04-25T14:06:56.034 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T14:06:56.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.148+0000 7f724723c640 1 Processor -- start 2024-04-25T14:06:56.148 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.148+0000 7f724723c640 1 -- start start 2024-04-25T14:06:56.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f724006cfd0 0x7f724006e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f724015d700 0x7f724015db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.150 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015e050 0x7f724015efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f724015eb10 con 0x7f724015d700 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f724006d3e0 con 0x7f724006cfd0 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f724723c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f724006e760 con 0x7f724015e050 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f724015d700 0x7f724015db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f724015d700 0x7f724015db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47832/0 (socket says 172.21.15.150:47832) 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 -- 172.21.15.150:0/1520285266 learned_addr learned my addr 172.21.15.150:0/1520285266 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f72457d9640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015e050 0x7f724015efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f7244fd8640 1 --2- 172.21.15.150:0/1520285266 >> v2:172.21.15.187:3300/0 conn(0x7f724006cfd0 0x7f724006e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 -- 172.21.15.150:0/1520285266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015e050 msgr2=0x7f724015efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015e050 0x7f724015efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 -- 172.21.15.150:0/1520285266 >> v2:172.21.15.187:3300/0 conn(0x7f724006cfd0 msgr2=0x7f724006e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.149+0000 7f723ffff640 1 --2- 172.21.15.150:0/1520285266 >> v2:172.21.15.187:3300/0 conn(0x7f724006cfd0 0x7f724006e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.150+0000 7f723ffff640 1 -- 172.21.15.150:0/1520285266 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724006d660 con 0x7f724015d700 2024-04-25T14:06:56.151 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.150+0000 7f723ffff640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724015d700 0x7f724015db10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7240074380 tx=0x7f723402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=521ac3a3eddad439 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:56.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.150+0000 7f723f7fe640 1 -- 172.21.15.150:0/1520285266 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234037070 con 0x7f724015d700 2024-04-25T14:06:56.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.150+0000 7f723f7fe640 1 -- 172.21.15.150:0/1520285266 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7234032440 con 0x7f724015d700 2024-04-25T14:06:56.152 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.150+0000 7f723f7fe640 1 -- 172.21.15.150:0/1520285266 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7234032810 con 0x7f724015d700 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 -- 172.21.15.150:0/1520285266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724015d700 msgr2=0x7f724015db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724015d700 0x7f724015db10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7240074380 tx=0x7f723402a140 comp rx=0 tx=0).stop 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 -- 172.21.15.150:0/1520285266 shutdown_connections 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015e050 0x7f724015efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 --2- 172.21.15.150:0/1520285266 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724015d700 0x7f724015db10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 --2- 172.21.15.150:0/1520285266 >> v2:172.21.15.187:3300/0 conn(0x7f724006cfd0 0x7f724006e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 -- 172.21.15.150:0/1520285266 >> 172.21.15.150:0/1520285266 conn(0x7f7240073dd0 msgr2=0x7f724009b630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 -- 172.21.15.150:0/1520285266 shutdown_connections 2024-04-25T14:06:56.153 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.151+0000 7f724723c640 1 -- 172.21.15.150:0/1520285266 wait complete. 2024-04-25T14:06:56.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.153+0000 7f724723c640 1 Processor -- start 2024-04-25T14:06:56.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.153+0000 7f724723c640 1 -- start start 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015d700 0x7f72401582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f724015e050 0x7f7240158ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f724015f4e0 con 0x7f724006cfd0 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f724015e7a0 con 0x7f724015e050 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f724723c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f724015f660 con 0x7f724015d700 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f7244fd8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f7244fd8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47846/0 (socket says 172.21.15.150:47846) 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f7244fd8640 1 -- 172.21.15.150:0/1954665316 learned_addr learned my addr 172.21.15.150:0/1954665316 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f723ffff640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015d700 0x7f72401582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.154+0000 7f72457d9640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f724015e050 0x7f7240158ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.156 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 -- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015d700 msgr2=0x7f72401582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015d700 0x7f72401582a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 -- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f724015e050 msgr2=0x7f7240158ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f724015e050 0x7f7240158ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 -- 172.21.15.150:0/1954665316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7240157200 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f7244fd8640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f723000fa40 tx=0x7f723000ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230004440 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7230009820 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72401574e0 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7230009bf0 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.155+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7240157aa0 con 0x7f724006cfd0 2024-04-25T14:06:56.157 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.156+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7240050740 con 0x7f724006cfd0 2024-04-25T14:06:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.157+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7230003610 con 0x7f724006cfd0 2024-04-25T14:06:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.157+0000 7f723d7fa640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 0x7f722c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.157+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5893+0+0 (secure 0 0 0) 0x7f7230097640 con 0x7f724006cfd0 2024-04-25T14:06:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.157+0000 7f723ffff640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 0x7f722c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:56.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.158+0000 7f723ffff640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 0x7f722c0769c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7234034000 tx=0x7f723402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:56.162 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.162+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f72300616c0 con 0x7f724006cfd0 2024-04-25T14:06:56.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:56.344+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f724006fce0 con 0x7f724006cfd0 2024-04-25T14:06:57.023 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.023+0000 7f723d7fa640 1 -- 172.21.15.150:0/1954665316 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v65) ==== 169+0+0 (secure 0 0 0) 0x7f72300659c0 con 0x7f724006cfd0 2024-04-25T14:06:57.024 INFO:teuthology.orchestra.run.smithi150.stderr:pool 'cephfs_data' removed 2024-04-25T14:06:57.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 msgr2=0x7f722c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 0x7f722c0769c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f7234034000 tx=0x7f723402e000 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 msgr2=0x7f7240153ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f723000fa40 tx=0x7f723000ff00 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 shutdown_connections 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f722c0744b0 0x7f722c0769c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f724015e050 0x7f7240158ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f724015d700 0x7f72401582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 --2- 172.21.15.150:0/1954665316 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f724006cfd0 0x7f7240153ad0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 >> 172.21.15.150:0/1954665316 conn(0x7f7240073dd0 msgr2=0x7f724006c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:57.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.027+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 shutdown_connections 2024-04-25T14:06:57.029 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.028+0000 7f724723c640 1 -- 172.21.15.150:0/1954665316 wait complete. 2024-04-25T14:06:57.045 DEBUG:teuthology.orchestra.run.smithi150:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:06:57.047 DEBUG:teuthology.orchestra.run.smithi187:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout:[global] 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: chdir = "" 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: auth supported = cephx 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: filestore xattr use omap = true 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: mon clock drift allowed = 1.000 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: osd crush chooseleaf type = 0 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: auth debug = true 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.088 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on old message = true 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on bug = true 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon pg warn max object skew = 0 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default size = 2 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow primary affinity = true 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd allow pg remap = true 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on no sortbitwise = false 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:06:57.089 INFO:teuthology.orchestra.run.smithi150.stdout: mon warn on too few osds = false 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: mon_allow_pool_size_one = true 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: osd default data pool replay window = 5 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: mon allow pool delete = true 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: mon cluster log file level = debug 2024-04-25T14:06:57.090 INFO:teuthology.orchestra.run.smithi150.stdout: debug asserts on shutdown = true 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: mon health detail to clog = false 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: ms die on skipped message = False 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout:[osd] 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd journal size = 100 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub load threshold = 5.0 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd scrub max interval = 600 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd recover clone overlap = true 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: osd recovery max chunk = 1048576 2024-04-25T14:06:57.091 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug shutdown = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug op order = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug verify stray on activate = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug trim objects = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd open classes on start = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug pg log writeout = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: osd map max advance = 10 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: journal zero on create = true 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:06:57.092 INFO:teuthology.orchestra.run.smithi150.stdout: filestore apply finisher threads = 3 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: bdev debug aio = true 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: osd debug misdirected ops = true 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore block size = 96636764160 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: bluestore fsck on mount = True 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluefs = 20 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: debug bluestore = 20 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: debug osd = 20 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: debug rocksdb = 10 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd full ratio = 0.9 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: osd objectstore = bluestore 2024-04-25T14:06:57.093 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout:[mgr] 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug mgr = 20 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: mgr/telemetry/nag = false 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout:[mon] 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:06:57.094 INFO:teuthology.orchestra.run.smithi150.stdout: debug mon = 20 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: debug paxos = 20 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: debug auth = 20 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon data avail warn = 5 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon mgr mkfs grace = 240 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd reporter subtree level = osd 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon osd prime pg temp = true 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:06:57.095 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: # 1m isn't quite enough 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: mon down mkfs grace = 300 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: mon op complaint time = 120 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout:[client] 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: rgw cache enabled = true 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable ops log = true 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: rgw enable usage log = true 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:06:57.096 INFO:teuthology.orchestra.run.smithi150.stdout: client mount timeout = 600 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: debug client = 20 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: pid file = "" 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.a] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.b] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mon.c] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.a] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.c] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.b] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mds.d] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout:[mds] 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds = 20 2024-04-25T14:06:57.097 INFO:teuthology.orchestra.run.smithi150.stdout: debug mds balancer = 20 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: debug ms = 1 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug frag = True 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: mds debug scatterstat = True 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: mds op complaint time = 180 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: mds verify scatter = True 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: osd op complaint time = 180 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: rados mon op timeout = 900 2024-04-25T14:06:57.098 INFO:teuthology.orchestra.run.smithi150.stdout: rados osd op timeout = 900 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout:[global] 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: chdir = "" 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: auth supported = cephx 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: filestore xattr use omap = true 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: mon clock drift allowed = 1.000 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: osd crush chooseleaf type = 0 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: auth debug = true 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.099 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on old message = true 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on bug = true 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon pg warn max object skew = 0 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: # disable pg_autoscaler by default for new pools 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default size = 2 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow primary affinity = true 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd allow pg remap = true 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on legacy crush tunables = false 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on crush straw calc version zero = false 2024-04-25T14:06:57.100 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on no sortbitwise = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on osd down out interval zero = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon warn on too few osds = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon_allow_pool_size_one = true 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: osd default data pool replay window = 5 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon allow pool delete = true 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon cluster log file level = debug 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: debug asserts on shutdown = true 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon health detail to clog = false 2024-04-25T14:06:57.101 INFO:teuthology.orchestra.run.smithi187.stdout: mon host = "172.21.15.150,172.21.15.187,[v2:172.21.15.187:3301,v1:172.21.15.187:6790]" 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: ms die on skipped message = False 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: fsid = 77163e42-55ae-4cab-b506-bd8c531cc087 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout:[osd] 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd journal size = 100 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub load threshold = 5.0 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd scrub max interval = 600 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd mclock profile = high_recovery_ops 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd recover clone overlap = true 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd recovery max chunk = 1048576 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.102 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug shutdown = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug op order = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug verify stray on activate = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug trim objects = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd open classes on start = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug pg log writeout = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd deep scrub update digest min age = 30 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: osd map max advance = 10 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: journal zero on create = true 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: filestore ondisk finisher threads = 3 2024-04-25T14:06:57.103 INFO:teuthology.orchestra.run.smithi187.stdout: filestore apply finisher threads = 3 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: bdev debug aio = true 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: osd debug misdirected ops = true 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore block size = 96636764160 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: bluestore fsck on mount = True 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluefs = 20 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: debug bluestore = 20 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: debug osd = 20 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: debug rocksdb = 10 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd backfillfull_ratio = 0.85 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd full ratio = 0.9 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd nearfull ratio = 0.8 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: osd failsafe full ratio = 0.95 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: osd objectstore = bluestore 2024-04-25T14:06:57.104 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout:[mgr] 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug mgr = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: mgr/telemetry/nag = false 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout:[mon] 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug mon = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug paxos = 20 2024-04-25T14:06:57.105 INFO:teuthology.orchestra.run.smithi187.stdout: debug auth = 20 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon data avail warn = 5 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon mgr mkfs grace = 240 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min pgs per osd = 4 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd reporter subtree level = osd 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon osd prime pg temp = true 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon reweight min bytes per osd = 10 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: auth mon ticket ttl = 660 # 11m 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: auth service ticket ttl = 240 # 4m 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: # don't complain about insecure global_id in the test suite 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-25T14:06:57.106 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: # 1m isn't quite enough 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: mon_down_mkfs_grace = 2m 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: mon_warn_on_filestore_osds = false 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: mon down mkfs grace = 300 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: mon op complaint time = 120 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout:[client] 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: rgw cache enabled = true 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable ops log = true 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: rgw enable usage log = true 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: client mount timeout = 600 2024-04-25T14:06:57.107 INFO:teuthology.orchestra.run.smithi187.stdout: debug client = 20 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: pid file = "" 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.a] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.b] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mon.c] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.a] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.c] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.b] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mds.d] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout:[mds] 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds = 20 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: debug mds balancer = 20 2024-04-25T14:06:57.108 INFO:teuthology.orchestra.run.smithi187.stdout: debug ms = 1 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug frag = True 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: mds debug scatterstat = True 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: mds op complaint time = 180 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: mds verify scatter = True 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: osd op complaint time = 180 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: rados mon op timeout = 900 2024-04-25T14:06:57.109 INFO:teuthology.orchestra.run.smithi187.stdout: rados osd op timeout = 900 2024-04-25T14:06:57.110 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-25T14:06:57.263 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.263+0000 7f9afd76c640 1 Processor -- start 2024-04-25T14:06:57.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.263+0000 7f9afd76c640 1 -- start start 2024-04-25T14:06:57.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.265+0000 7f9afd76c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f9af80ce4d0 0x7f9af8158760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.265+0000 7f9afd76c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9af8161dd0 0x7f9af815f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9afd76c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9af8158ca0 con 0x7f9af815f550 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9afd76c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f9af8162620 con 0x7f9af8161dd0 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9afd76c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f9af8162900 con 0x7f9af80ce4d0 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af67fc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f9af8161dd0 0x7f9af815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af77fe640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f9af815f550 0x7f9af80d1f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:37526/0 (socket says 172.21.15.150:37526) 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af77fe640 1 -- 172.21.15.150:0/2831472828 learned_addr learned my addr 172.21.15.150:0/2831472828 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af6ffd640 1 --2- 172.21.15.150:0/2831472828 >> v2:172.21.15.187:3300/0 conn(0x7f9af80ce4d0 0x7f9af8158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af67fc640 1 -- 172.21.15.150:0/2831472828 >> v2:172.21.15.187:3300/0 conn(0x7f9af80ce4d0 msgr2=0x7f9af8158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af67fc640 1 --2- 172.21.15.150:0/2831472828 >> v2:172.21.15.187:3300/0 conn(0x7f9af80ce4d0 0x7f9af8158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af67fc640 1 -- 172.21.15.150:0/2831472828 >> v1:172.21.15.150:6789/0 conn(0x7f9af815f550 legacy=0x7f9af80d1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af67fc640 1 -- 172.21.15.150:0/2831472828 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af8162be0 con 0x7f9af8161dd0 2024-04-25T14:06:57.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.266+0000 7f9af5ffb640 1 -- 172.21.15.150:0/2831472828 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3784168833 0 0) 0x7f9af8158ca0 con 0x7f9af815f550 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.267+0000 7f9af67fc640 1 --2- 172.21.15.150:0/2831472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af8161dd0 0x7f9af815f010 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9af80d18b0 tx=0x7f9aec02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af5c4f38ed4ecc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.267+0000 7f9af5ffb640 1 -- 172.21.15.150:0/2831472828 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec037070 con 0x7f9af8161dd0 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.267+0000 7f9af5ffb640 1 -- 172.21.15.150:0/2831472828 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec032440 con 0x7f9af8161dd0 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.267+0000 7f9af5ffb640 1 -- 172.21.15.150:0/2831472828 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec032810 con 0x7f9af8161dd0 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.268+0000 7f9afd76c640 1 -- 172.21.15.150:0/2831472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af8161dd0 msgr2=0x7f9af815f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.268+0000 7f9afd76c640 1 --2- 172.21.15.150:0/2831472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af8161dd0 0x7f9af815f010 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9af80d18b0 tx=0x7f9aec02a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.268+0000 7f9afd76c640 1 -- 172.21.15.150:0/2831472828 shutdown_connections 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.268+0000 7f9afd76c640 1 --2- 172.21.15.150:0/2831472828 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af8161dd0 0x7f9af815f010 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.268+0000 7f9afd76c640 1 --2- 172.21.15.150:0/2831472828 >> v2:172.21.15.187:3300/0 conn(0x7f9af80ce4d0 0x7f9af8158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.269+0000 7f9afd76c640 1 -- 172.21.15.150:0/2831472828 >> 172.21.15.150:0/2831472828 conn(0x7f9af80d03d0 msgr2=0x7f9af80d07e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.269+0000 7f9afd76c640 1 -- 172.21.15.150:0/2831472828 shutdown_connections 2024-04-25T14:06:57.269 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.269+0000 7f9afd76c640 1 -- 172.21.15.150:0/2831472828 wait complete. 2024-04-25T14:06:57.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.271+0000 7f9afd76c640 1 Processor -- start 2024-04-25T14:06:57.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.271+0000 7f9afd76c640 1 -- start start 2024-04-25T14:06:57.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af80ce4d0 0x7f9af81f89d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9af8161dd0 0x7f9af8151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f9af81621e0 con 0x7f9af815f550 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f9af815e790 con 0x7f9af80ce4d0 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9afd76c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f9af8162360 con 0x7f9af8161dd0 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:47868/0 (socket says 172.21.15.150:47868) 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 -- 172.21.15.150:0/4128410323 learned_addr learned my addr 172.21.15.150:0/4128410323 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af6ffd640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af80ce4d0 0x7f9af81f89d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af77fe640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9af8161dd0 0x7f9af8151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 -- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9af8161dd0 msgr2=0x7f9af8151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9af8161dd0 0x7f9af8151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 -- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af80ce4d0 msgr2=0x7f9af81f89d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af80ce4d0 0x7f9af81f89d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.272+0000 7f9af67fc640 1 -- 172.21.15.150:0/4128410323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af81f5e50 con 0x7f9af815f550 2024-04-25T14:06:57.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9af67fc640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9aec035040 tx=0x7f9aec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec037050 con 0x7f9af815f550 2024-04-25T14:06:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af8150890 con 0x7f9af815f550 2024-04-25T14:06:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af81f6410 con 0x7f9af815f550 2024-04-25T14:06:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aec038040 con 0x7f9af815f550 2024-04-25T14:06:57.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.273+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aec004450 con 0x7f9af815f550 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.274+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ad8005290 con 0x7f9af815f550 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.275+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f9aec03c6d0 con 0x7f9af815f550 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.275+0000 7f9adf7fe640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 0x7f9ae40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.275+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f9aec0b5960 con 0x7f9af815f550 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.275+0000 7f9af6ffd640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 0x7f9ae40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.276 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.276+0000 7f9af6ffd640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 0x7f9ae40769c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9af80ce290 tx=0x7f9ae8009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.280 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.280+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9aec04c020 con 0x7f9af815f550 2024-04-25T14:06:57.455 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.455+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f9ad8005850 con 0x7f9af815f550 2024-04-25T14:06:57.458 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.459+0000 7f9adf7fe640 1 -- 172.21.15.150:0/4128410323 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f9aec07fbd0 con 0x7f9af815f550 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 msgr2=0x7f9ae40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 0x7f9ae40769c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9af80ce290 tx=0x7f9ae8009190 comp rx=0 tx=0).stop 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 msgr2=0x7f9af8150350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f9aec035040 tx=0x7f9aec0046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 shutdown_connections 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f9ae40744b0 0x7f9ae40769c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.461 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f9af8161dd0 0x7f9af8151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f9af815f550 0x7f9af8150350 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 --2- 172.21.15.150:0/4128410323 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f9af80ce4d0 0x7f9af81f89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 >> 172.21.15.150:0/4128410323 conn(0x7f9af80d03d0 msgr2=0x7f9af8160c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:57.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 shutdown_connections 2024-04-25T14:06:57.462 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.461+0000 7f9afd76c640 1 -- 172.21.15.150:0/4128410323 wait complete. 2024-04-25T14:06:57.472 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters' 2024-04-25T14:06:57.580 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.580+0000 7fe0c59a8640 1 Processor -- start 2024-04-25T14:06:57.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.580+0000 7fe0c59a8640 1 -- start start 2024-04-25T14:06:57.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.581+0000 7fe0c59a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0c006d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.581+0000 7fe0c59a8640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fe0c006d830 0x7fe0c0163ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.581+0000 7fe0c59a8640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0c006c570 con 0x7fe0c01601d0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.581+0000 7fe0c59a8640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fe0c01605e0 con 0x7fe0c006d830 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.581+0000 7fe0c59a8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c00d5980 con 0x7fe0c00b9070 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0beffd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fe0c01601d0 0x7fe0c00d5100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46394/0 (socket says 172.21.15.150:46394) 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0beffd640 1 -- 172.21.15.150:0/2215031307 learned_addr learned my addr 172.21.15.150:0/2215031307 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0c006d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bf7fe640 1 --2- 172.21.15.150:0/2215031307 >> v2:172.21.15.187:3300/0 conn(0x7fe0c006d830 0x7fe0c0163ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1610926730 0 0) 0x7fe0c006c570 con 0x7fe0c01601d0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0a8003520 con 0x7fe0c01601d0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2354914614 0 0) 0x7fe0a8003520 con 0x7fe0c01601d0 2024-04-25T14:06:57.590 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 msgr2=0x7fe0c006d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0c006d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 >> v2:172.21.15.187:3300/0 conn(0x7fe0c006d830 msgr2=0x7fe0c0163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 --2- 172.21.15.150:0/2215031307 >> v2:172.21.15.187:3300/0 conn(0x7fe0c006d830 0x7fe0c0163ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c006c810 con 0x7fe0c01601d0 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fe0b0001670 con 0x7fe0c01601d0 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bf7fe640 1 --2- 172.21.15.150:0/2215031307 >> v2:172.21.15.187:3300/0 conn(0x7fe0c006d830 0x7fe0c0163ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0c006d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 >> v1:172.21.15.150:6789/0 conn(0x7fe0c01601d0 legacy=0x7fe0c00d5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0a80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a8003f10 0x7fe0a8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8004d40 0x7fe0a8005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c01605e0 con 0x7fe0a8003f10 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c00d5980 con 0x7fe0a8004d40 2024-04-25T14:06:57.591 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0a800baa0 con 0x7fe0c00b9070 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a8003f10 0x7fe0a8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0beffd640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0a80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.582+0000 7fe0bf7fe640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8004d40 0x7fe0a8005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 -- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 msgr2=0x7fe0a80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0c00b9070 0x7fe0a80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 -- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8004d40 msgr2=0x7fe0a8005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0a8004d40 0x7fe0a8005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 -- 172.21.15.150:0/2215031307 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a8003520 con 0x7fe0a8003f10 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a8003f10 0x7fe0a8004800 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009c60 tx=0x7fe0a8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555b19224a3349a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.583+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4036070 con 0x7fe0a8003f10 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b4031400 con 0x7fe0a8003f10 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0bdffb640 1 -- 172.21.15.150:0/2215031307 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4009860 con 0x7fe0a8003f10 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 -- 172.21.15.150:0/2215031307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a8003f10 msgr2=0x7fe0a8004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.592 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/2215031307 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0a8003f10 0x7fe0a8004800 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009c60 tx=0x7fe0a8005830 comp rx=0 tx=0).stop 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0bf7fe640 1 -- 172.21.15.150:0/2215031307 reap_dead start 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 -- 172.21.15.150:0/2215031307 shutdown_connections 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 -- 172.21.15.150:0/2215031307 >> 172.21.15.150:0/2215031307 conn(0x7fe0c00d5d90 msgr2=0x7fe0c00d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 -- 172.21.15.150:0/2215031307 shutdown_connections 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.584+0000 7fe0c59a8640 1 -- 172.21.15.150:0/2215031307 wait complete. 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.586+0000 7fe0c59a8640 1 Processor -- start 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.586+0000 7fe0c59a8640 1 -- start start 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a8003f10 0x7fe0c00b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c00aea00 0x7fe0c00cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c00b9480 con 0x7fe0c01f7c90 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c006cdd0 con 0x7fe0c00aea00 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.587+0000 7fe0c59a8640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fe0c00b9600 con 0x7fe0a8003f10 2024-04-25T14:06:57.593 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33456/0 (socket says 172.21.15.150:33456) 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 -- 172.21.15.150:0/1903980294 learned_addr learned my addr 172.21.15.150:0/1903980294 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0bf7fe640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c00aea00 0x7fe0c00cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0beffd640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a8003f10 0x7fe0c00b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 -- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a8003f10 msgr2=0x7fe0c00b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a8003f10 0x7fe0c00b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 -- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c00aea00 msgr2=0x7fe0c00cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c00aea00 0x7fe0c00cefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 -- 172.21.15.150:0/1903980294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c00b0890 con 0x7fe0c01f7c90 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0be7fc640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009d80 tx=0x7fe0b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.588+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4040020 con 0x7fe0c01f7c90 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.589+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0b403b460 con 0x7fe0c01f7c90 2024-04-25T14:06:57.594 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.589+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c00ad960 con 0x7fe0c01f7c90 2024-04-25T14:06:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.589+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b4036040 con 0x7fe0c01f7c90 2024-04-25T14:06:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.589+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c00adf20 con 0x7fe0c01f7c90 2024-04-25T14:06:57.595 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.589+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0c0076a40 con 0x7fe0c01f7c90 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.590+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fe0b4043020 con 0x7fe0c01f7c90 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.590+0000 7fe0a7fff640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 0x7fe0ac0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.591+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fe0b40b53c0 con 0x7fe0c01f7c90 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.591+0000 7fe0beffd640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 0x7fe0ac0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.591+0000 7fe0beffd640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 0x7fe0ac0769f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe0c008df30 tx=0x7fe0b00078f0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:57.596 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.595+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe0b404a070 con 0x7fe0c01f7c90 2024-04-25T14:06:57.839 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:57.839+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters"]} v 0) -- 0x7fe0c015f930 con 0x7fe0c01f7c90 2024-04-25T14:06:58.024 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.024+0000 7fe0a7fff640 1 -- 172.21.15.150:0/1903980294 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_openfiletable.OpenFileTable.test_perf_counters"]}]=0 v232) ==== 143+0+0 (secure 0 0 0) 0x7fe0b407f6b0 con 0x7fe0c01f7c90 2024-04-25T14:06:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 msgr2=0x7fe0ac0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 0x7fe0ac0769f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe0c008df30 tx=0x7fe0b00078f0 comp rx=0 tx=0).stop 2024-04-25T14:06:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 msgr2=0x7fe0c015e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:58.027 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fe0b4009d80 tx=0x7fe0b4004730 comp rx=0 tx=0).stop 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 shutdown_connections 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fe0ac0744e0 0x7fe0ac0769f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fe0c00aea00 0x7fe0c00cefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fe0c01f7c90 0x7fe0c015e060 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 --2- 172.21.15.150:0/1903980294 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fe0a8003f10 0x7fe0c00b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 >> 172.21.15.150:0/1903980294 conn(0x7fe0c00d5d90 msgr2=0x7fe0c015ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 shutdown_connections 2024-04-25T14:06:58.028 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.027+0000 7fe0c59a8640 1 -- 172.21.15.150:0/1903980294 wait complete. 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner:Opening a file should increment omap_total_updates by 1. ... ok 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner: 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner:Ran 2 tests in 241.067s 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner: 2024-04-25T14:06:58.039 INFO:tasks.cephfs_test_runner:OK 2024-04-25T14:06:58.040 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-25T14:06:58.049 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T14:06:58.072 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T14:06:58.072 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:58.072 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:58.087 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:06:58.087 DEBUG:teuthology.orchestra.run.smithi150:> set -ex 2024-04-25T14:06:58.087 DEBUG:teuthology.orchestra.run.smithi150:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:58.143 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056109,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1362 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-25T14:06:58.143 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:06:58.143 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:58.162 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:06:58.162 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2024-04-25T14:06:58.162 DEBUG:teuthology.orchestra.run.smithi187:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T14:06:58.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056096,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1329 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-25T14:06:58.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.220 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:06:58.235 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-25T14:06:58.235 INFO:teuthology.orchestra.run.smithi150.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T14:06:58.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.236 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-25T14:06:58.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.300 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T14:06:58.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.365 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-04-25T14:06:58.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.460 DEBUG:teuthology.orchestra.run.smithi150:> ip netns list 2024-04-25T14:06:58.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.475 DEBUG:teuthology.orchestra.run.smithi150:> sudo ip link delete ceph-brx 2024-04-25T14:06:58.541 INFO:teuthology.orchestra.run.smithi150.stderr:Cannot find device "ceph-brx" 2024-04-25T14:06:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:06:58.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.543 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:06:58.561 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-25T14:06:58.561 INFO:teuthology.orchestra.run.smithi187.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-25T14:06:58.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.561 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-25T14:06:58.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.630 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-25T14:06:58.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.697 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-25T14:06:58.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.794 DEBUG:teuthology.orchestra.run.smithi187:> ip netns list 2024-04-25T14:06:58.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T14:06:58.810 DEBUG:teuthology.orchestra.run.smithi187:> sudo ip link delete ceph-brx 2024-04-25T14:06:58.877 INFO:teuthology.orchestra.run.smithi187.stderr:Cannot find device "ceph-brx" 2024-04-25T14:06:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:06:58.878 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-25T14:06:58.887 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-25T14:06:58.888 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:06:58.999 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.999+0000 7f61978e9640 1 Processor -- start 2024-04-25T14:06:59.000 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:58.999+0000 7f61978e9640 1 -- start start 2024-04-25T14:06:59.001 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f61978e9640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61900ae7e0 0x7f61900aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f61978e9640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61900b4750 con 0x7f61900b57f0 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f61978e9640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61900b4a30 con 0x7f619015df00 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f61978e9640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f619015e310 con 0x7f61900ae7e0 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6195e86640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f61900ae7e0 0x7f61900aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6194e84640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f61900b57f0 0x7f61900b5c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46396/0 (socket says 172.21.15.150:46396) 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6194e84640 1 -- 172.21.15.150:0/1899279494 learned_addr learned my addr 172.21.15.150:0/1899279494 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6195e86640 1 -- 172.21.15.150:0/1899279494 >> v1:172.21.15.187:6789/0 conn(0x7f619015df00 legacy=0x7f619006f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6195e86640 1 -- 172.21.15.150:0/1899279494 >> v1:172.21.15.150:6789/0 conn(0x7f61900b57f0 legacy=0x7f61900b5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.002 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6187fff640 1 -- 172.21.15.150:0/1899279494 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256967998 0 0) 0x7f61900b4750 con 0x7f61900b57f0 2024-04-25T14:06:59.003 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.001+0000 7f6195e86640 1 -- 172.21.15.150:0/1899279494 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61900b4d10 con 0x7f61900ae7e0 2024-04-25T14:06:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.002+0000 7f6195e86640 1 --2- 172.21.15.150:0/1899279494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900ae7e0 0x7f61900aebf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f61880097a0 tx=0x7f618802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4559bb6df6dac588 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.002+0000 7f6187fff640 1 -- 172.21.15.150:0/1899279494 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188037070 con 0x7f61900ae7e0 2024-04-25T14:06:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.002+0000 7f6187fff640 1 -- 172.21.15.150:0/1899279494 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6188032440 con 0x7f61900ae7e0 2024-04-25T14:06:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.002+0000 7f6187fff640 1 -- 172.21.15.150:0/1899279494 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6188032810 con 0x7f61900ae7e0 2024-04-25T14:06:59.004 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 -- 172.21.15.150:0/1899279494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900ae7e0 msgr2=0x7f61900aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 --2- 172.21.15.150:0/1899279494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900ae7e0 0x7f61900aebf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f61880097a0 tx=0x7f618802a1a0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 -- 172.21.15.150:0/1899279494 shutdown_connections 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 --2- 172.21.15.150:0/1899279494 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900ae7e0 0x7f61900aebf0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 -- 172.21.15.150:0/1899279494 >> 172.21.15.150:0/1899279494 conn(0x7f61900d1ae0 msgr2=0x7f61900d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 -- 172.21.15.150:0/1899279494 shutdown_connections 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.003+0000 7f61978e9640 1 -- 172.21.15.150:0/1899279494 wait complete. 2024-04-25T14:06:59.005 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.005+0000 7f61978e9640 1 Processor -- start 2024-04-25T14:06:59.006 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.005+0000 7f61978e9640 1 -- start start 2024-04-25T14:06:59.007 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61900ae7e0 0x7f61901e65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f619015df00 0x7f61900cfff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f619006f950 con 0x7f61900b57f0 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f61900b5360 con 0x7f619015df00 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f61978e9640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f61900af130 con 0x7f61900ae7e0 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33474/0 (socket says 172.21.15.150:33474) 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 -- 172.21.15.150:0/3153918492 learned_addr learned my addr 172.21.15.150:0/3153918492 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 -- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61900ae7e0 msgr2=0x7f61901e65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61900ae7e0 0x7f61901e65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 -- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f619015df00 msgr2=0x7f61900cfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f619015df00 0x7f61900cfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.008 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.006+0000 7f6194e84640 1 -- 172.21.15.150:0/3153918492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61901e6ec0 con 0x7f61900b57f0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f6194e84640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f618000c3f0 tx=0x7f618000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61800074e0 con 0x7f61900b57f0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61800047e0 con 0x7f61900b57f0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61901e71a0 con 0x7f61900b57f0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180004c10 con 0x7f61900b57f0 2024-04-25T14:06:59.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.007+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61901e7760 con 0x7f61900b57f0 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.008+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6190076a40 con 0x7f61900b57f0 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.009+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6180011070 con 0x7f61900b57f0 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.009+0000 7f6185ffb640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 0x7f61740769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.009+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f6180094cd0 con 0x7f61900b57f0 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.009+0000 7f6195685640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 0x7f61740769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.009+0000 7f6195685640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 0x7f61740769f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f618c0012e0 tx=0x7f618c007d30 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.014+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f618005f580 con 0x7f61900b57f0 2024-04-25T14:06:59.183 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.183+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61900b6860 con 0x7f61740744e0 2024-04-25T14:06:59.184 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.184+0000 7f6185ffb640 1 -- 172.21.15.150:0/3153918492 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13453 (secure 0 0 0) 0x7f61900b6860 con 0x7f61740744e0 2024-04-25T14:06:59.184 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:59.185 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:06:59.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.186+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 msgr2=0x7f61740769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.186+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 0x7f61740769f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f618c0012e0 tx=0x7f618c007d30 comp rx=0 tx=0).stop 2024-04-25T14:06:59.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.186+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 msgr2=0x7f61901f1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.186+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f618000c3f0 tx=0x7f618000c8b0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 shutdown_connections 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f61740744e0 0x7f61740769f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f619015df00 0x7f61900cfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f61900b57f0 0x7f61901f1dc0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 --2- 172.21.15.150:0/3153918492 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f61900ae7e0 0x7f61901e65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 >> 172.21.15.150:0/3153918492 conn(0x7f61900d1ae0 msgr2=0x7f619015f850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 shutdown_connections 2024-04-25T14:06:59.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.187+0000 7f61978e9640 1 -- 172.21.15.150:0/3153918492 wait complete. 2024-04-25T14:06:59.198 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-25T14:06:58.659299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223312,"kb_used_data":7148,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751408,"statfs":{"total":773094113280,"available":772865441792,"internally_reserved":0,"allocated":7319552,"data_stored":3996882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7290,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-79,"num_read_kb":-83,"num_write":-74,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001412"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-25T14:06:56.024398+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:06:56.024398+0000","last_peered":"2024-04-25T14:06:56.024398+0000","last_clean":"2024-04-25T14:06:56.024398+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:06:56.024398+0000","last_undegraded":"2024-04-25T14:06:56.024398+0000","last_fullsized":"2024-04-25T14:06:56.024398+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":939056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":348731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1363968,"data_stored":945471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:06:59.199 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:06:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.307+0000 7fa840420640 1 Processor -- start 2024-04-25T14:06:59.307 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.307+0000 7fa840420640 1 -- start start 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa840420640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8381601d0 0x7fa8380d5100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa840420640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa8380b9070 0x7fa83806d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa840420640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa83806c570 con 0x7fa83806d830 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa840420640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa8381605e0 con 0x7fa8381601d0 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa840420640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa8380d5980 con 0x7fa8380b9070 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa83e1bc640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa8381601d0 0x7fa8380d5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa83e9bd640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fa83806d830 0x7fa8380d0710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46410/0 (socket says 172.21.15.150:46410) 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.308+0000 7fa83e9bd640 1 -- 172.21.15.150:0/4026122646 learned_addr learned my addr 172.21.15.150:0/4026122646 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83d9bb640 1 --2- 172.21.15.150:0/4026122646 >> v2:172.21.15.187:3300/0 conn(0x7fa8380b9070 0x7fa83806d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83e1bc640 1 -- 172.21.15.150:0/4026122646 >> v2:172.21.15.187:3300/0 conn(0x7fa8380b9070 msgr2=0x7fa83806d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83e1bc640 1 --2- 172.21.15.150:0/4026122646 >> v2:172.21.15.187:3300/0 conn(0x7fa8380b9070 0x7fa83806d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.309 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83e1bc640 1 -- 172.21.15.150:0/4026122646 >> v1:172.21.15.150:6789/0 conn(0x7fa83806d830 legacy=0x7fa8380d0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83d1ba640 1 -- 172.21.15.150:0/4026122646 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4239651789 0 0) 0x7fa83806c570 con 0x7fa83806d830 2024-04-25T14:06:59.310 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83e1bc640 1 -- 172.21.15.150:0/4026122646 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83806c810 con 0x7fa8381601d0 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.309+0000 7fa83e1bc640 1 --2- 172.21.15.150:0/4026122646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8381601d0 0x7fa8380d5100 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa8380b5790 tx=0x7fa83402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b271c349cbf0ce4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa83d1ba640 1 -- 172.21.15.150:0/4026122646 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834037070 con 0x7fa8381601d0 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa83d1ba640 1 -- 172.21.15.150:0/4026122646 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa834032440 con 0x7fa8381601d0 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa83d1ba640 1 -- 172.21.15.150:0/4026122646 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa83402ae60 con 0x7fa8381601d0 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 -- 172.21.15.150:0/4026122646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8381601d0 msgr2=0x7fa8380d5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 --2- 172.21.15.150:0/4026122646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8381601d0 0x7fa8380d5100 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa8380b5790 tx=0x7fa83402a110 comp rx=0 tx=0).stop 2024-04-25T14:06:59.311 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 -- 172.21.15.150:0/4026122646 shutdown_connections 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 --2- 172.21.15.150:0/4026122646 >> v2:172.21.15.187:3300/0 conn(0x7fa8380b9070 0x7fa83806d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 --2- 172.21.15.150:0/4026122646 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa8381601d0 0x7fa8380d5100 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 -- 172.21.15.150:0/4026122646 >> 172.21.15.150:0/4026122646 conn(0x7fa8380d5d90 msgr2=0x7fa8380d61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 -- 172.21.15.150:0/4026122646 shutdown_connections 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.310+0000 7fa840420640 1 -- 172.21.15.150:0/4026122646 wait complete. 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.312+0000 7fa840420640 1 Processor -- start 2024-04-25T14:06:59.312 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.312+0000 7fa840420640 1 -- start start 2024-04-25T14:06:59.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 0x7fa8380ac700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.315 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 0x7fa8381fa460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa8380d5640 con 0x7fa83806d830 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa83806cdd0 con 0x7fa8381601d0 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.313+0000 7fa840420640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa8380b9480 con 0x7fa8380b9070 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33494/0 (socket says 172.21.15.150:33494) 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 -- 172.21.15.150:0/2373245211 learned_addr learned my addr 172.21.15.150:0/2373245211 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83d9bb640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 0x7fa8380ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e9bd640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 0x7fa8381fa460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 -- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 msgr2=0x7fa8380ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 0x7fa8380ac700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 -- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 msgr2=0x7fa8381fa460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 0x7fa8381fa460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.316 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 -- 172.21.15.150:0/2373245211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8381f7540 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e9bd640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 0x7fa8381fa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83d9bb640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 0x7fa8380ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa83e1bc640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa834035040 tx=0x7fa8340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834037050 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa834038040 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa834006e10 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8381f6f80 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.314+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8381f9c60 con 0x7fa83806d830 2024-04-25T14:06:59.317 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.315+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa838076a40 con 0x7fa83806d830 2024-04-25T14:06:59.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.316+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa834030980 con 0x7fa83806d830 2024-04-25T14:06:59.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.316+0000 7fa82effd640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 0x7fa81c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.317+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fa8340b59c0 con 0x7fa83806d830 2024-04-25T14:06:59.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.317+0000 7fa83d9bb640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 0x7fa81c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.318 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.317+0000 7fa83d9bb640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 0x7fa81c0769f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8380d0d90 tx=0x7fa828008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.321 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.321+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8340b7e20 con 0x7fa83806d830 2024-04-25T14:06:59.492 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.492+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa8380c9660 con 0x7fa81c0744e0 2024-04-25T14:06:59.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.493+0000 7fa82effd640 1 -- 172.21.15.150:0/2373245211 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13453 (secure 0 0 0) 0x7fa8380c9660 con 0x7fa81c0744e0 2024-04-25T14:06:59.493 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:59.493 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:06:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 msgr2=0x7fa81c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 0x7fa81c0769f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8380d0d90 tx=0x7fa828008040 comp rx=0 tx=0).stop 2024-04-25T14:06:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 msgr2=0x7fa8381f65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa834035040 tx=0x7fa8340046c0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 shutdown_connections 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa81c0744e0 0x7fa81c0769f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa8381601d0 0x7fa8381fa460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa8380b9070 0x7fa8380ac700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 --2- 172.21.15.150:0/2373245211 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa83806d830 0x7fa8381f65c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 >> 172.21.15.150:0/2373245211 conn(0x7fa8380d5d90 msgr2=0x7fa83815ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 shutdown_connections 2024-04-25T14:06:59.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.495+0000 7fa840420640 1 -- 172.21.15.150:0/2373245211 wait complete. 2024-04-25T14:06:59.506 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-25T14:06:58.659299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223312,"kb_used_data":7148,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751408,"statfs":{"total":773094113280,"available":772865441792,"internally_reserved":0,"allocated":7319552,"data_stored":3996882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7290,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-79,"num_read_kb":-83,"num_write":-74,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001412"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-25T14:06:56.024398+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:06:56.024398+0000","last_peered":"2024-04-25T14:06:56.024398+0000","last_clean":"2024-04-25T14:06:56.024398+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:06:56.024398+0000","last_undegraded":"2024-04-25T14:06:56.024398+0000","last_fullsized":"2024-04-25T14:06:56.024398+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":939056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":348731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1363968,"data_stored":945471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:06:59.507 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-25T14:06:59.507 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:06:59.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.614+0000 7f8e4529f640 1 Processor -- start 2024-04-25T14:06:59.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.614+0000 7f8e4529f640 1 -- start start 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e40163c40 0x7f8e400d08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 0x7f8e400d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 0x7f8e4006c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f8e400d55b0 con 0x7f8e40163c40 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f8e400d4510 con 0x7f8e400d0de0 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e4529f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e400d47f0 con 0x7f8e4006c570 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e40163c40 0x7f8e400d08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f8e40163c40 0x7f8e400d08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33508/0 (socket says 172.21.15.150:33508) 2024-04-25T14:06:59.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 -- 172.21.15.150:0/347488635 learned_addr learned my addr 172.21.15.150:0/347488635 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3e7fc640 1 --2- 172.21.15.150:0/347488635 >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 0x7f8e400d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3f7fe640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 0x7f8e4006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 -- 172.21.15.150:0/347488635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 msgr2=0x7f8e4006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 0x7f8e4006c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 -- 172.21.15.150:0/347488635 >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 msgr2=0x7f8e400d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 --2- 172.21.15.150:0/347488635 >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 0x7f8e400d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3effd640 1 -- 172.21.15.150:0/347488635 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e400d4ad0 con 0x7f8e40163c40 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3e7fc640 1 --2- 172.21.15.150:0/347488635 >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 0x7f8e400d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.616+0000 7f8e3f7fe640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 0x7f8e4006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.617+0000 7f8e3effd640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e40163c40 0x7f8e400d08a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8e400d3a70 tx=0x7f8e3002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe973d6cfc2a5fca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.617+0000 7f8e3dffb640 1 -- 172.21.15.150:0/347488635 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30036070 con 0x7f8e40163c40 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.617+0000 7f8e3dffb640 1 -- 172.21.15.150:0/347488635 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e30031600 con 0x7f8e40163c40 2024-04-25T14:06:59.618 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.617+0000 7f8e3dffb640 1 -- 172.21.15.150:0/347488635 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e300319d0 con 0x7f8e40163c40 2024-04-25T14:06:59.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 -- 172.21.15.150:0/347488635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e40163c40 msgr2=0x7f8e400d08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e40163c40 0x7f8e400d08a0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f8e400d3a70 tx=0x7f8e3002a610 comp rx=0 tx=0).stop 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 -- 172.21.15.150:0/347488635 shutdown_connections 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e4006c570 0x7f8e4006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 --2- 172.21.15.150:0/347488635 >> v2:172.21.15.187:3300/0 conn(0x7f8e400d0de0 0x7f8e400d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 --2- 172.21.15.150:0/347488635 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e40163c40 0x7f8e400d08a0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 -- 172.21.15.150:0/347488635 >> 172.21.15.150:0/347488635 conn(0x7f8e400b7650 msgr2=0x7f8e400b7a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 -- 172.21.15.150:0/347488635 shutdown_connections 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.618+0000 7f8e4529f640 1 -- 172.21.15.150:0/347488635 wait complete. 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.620+0000 7f8e4529f640 1 Processor -- start 2024-04-25T14:06:59.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.620+0000 7f8e4529f640 1 -- start start 2024-04-25T14:06:59.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e4006c570 0x7f8e400c9f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e40163c40 0x7f8e400ba0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400d41a0 con 0x7f8e400d0de0 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f8e400d5090 con 0x7f8e4006c570 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e4529f640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f8e4006cec0 con 0x7f8e40163c40 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33524/0 (socket says 172.21.15.150:33524) 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 -- 172.21.15.150:0/3118648942 learned_addr learned my addr 172.21.15.150:0/3118648942 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3f7fe640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e40163c40 0x7f8e400ba0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3effd640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e4006c570 0x7f8e400c9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 -- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e40163c40 msgr2=0x7f8e400ba0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e40163c40 0x7f8e400ba0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 -- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e4006c570 msgr2=0x7f8e400c9f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.621+0000 7f8e3e7fc640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e4006c570 0x7f8e400c9f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e3e7fc640 1 -- 172.21.15.150:0/3118648942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e400ca4c0 con 0x7f8e400d0de0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e3e7fc640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c7c0 tx=0x7f8e3400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34015070 con 0x7f8e400d0de0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e34007820 con 0x7f8e400d0de0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4015aab0 con 0x7f8e400d0de0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e34007bf0 con 0x7f8e400d0de0 2024-04-25T14:06:59.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.622+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e401eb820 con 0x7f8e400d0de0 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.623+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e400ca720 con 0x7f8e400d0de0 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.624+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f8e34018070 con 0x7f8e400d0de0 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.624+0000 7f8e277fe640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 0x7f8e2c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.624+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f8e34094ea0 con 0x7f8e400d0de0 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.624+0000 7f8e3effd640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 0x7f8e2c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.625+0000 7f8e3effd640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 0x7f8e2c0769c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e30033000 tx=0x7f8e30009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.629+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e3405f100 con 0x7f8e400d0de0 2024-04-25T14:06:59.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.799+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8e40076a40 con 0x7f8e2c0744b0 2024-04-25T14:06:59.799 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.799+0000 7f8e277fe640 1 -- 172.21.15.150:0/3118648942 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13453 (secure 0 0 0) 0x7f8e40076a40 con 0x7f8e2c0744b0 2024-04-25T14:06:59.800 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:06:59.800 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:06:59.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 msgr2=0x7f8e2c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 0x7f8e2c0769c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f8e30033000 tx=0x7f8e30009000 comp rx=0 tx=0).stop 2024-04-25T14:06:59.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 msgr2=0x7f8e4015a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f8e3400c7c0 tx=0x7f8e3400cc80 comp rx=0 tx=0).stop 2024-04-25T14:06:59.802 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 shutdown_connections 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f8e2c0744b0 0x7f8e2c0769c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f8e40163c40 0x7f8e400ba0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f8e400d0de0 0x7f8e4015a570 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 --2- 172.21.15.150:0/3118648942 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f8e4006c570 0x7f8e400c9f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 >> 172.21.15.150:0/3118648942 conn(0x7f8e400b7650 msgr2=0x7f8e401630f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 shutdown_connections 2024-04-25T14:06:59.803 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.802+0000 7f8e4529f640 1 -- 172.21.15.150:0/3118648942 wait complete. 2024-04-25T14:06:59.813 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-04-25T14:06:58.659299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223312,"kb_used_data":7148,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751408,"statfs":{"total":773094113280,"available":772865441792,"internally_reserved":0,"allocated":7319552,"data_stored":3996882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7290,"num_objects":-24,"num_object_clones":0,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":-79,"num_read_kb":-83,"num_write":-74,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001412"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-04-25T14:06:56.024398+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:06:56.024398+0000","last_peered":"2024-04-25T14:06:56.024398+0000","last_clean":"2024-04-25T14:06:56.024398+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:06:56.024398+0000","last_undegraded":"2024-04-25T14:06:56.024398+0000","last_fullsized":"2024-04-25T14:06:56.024398+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574913,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28344,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343496,"statfs":{"total":96636764160,"available":96607739904,"internally_reserved":0,"allocated":1355776,"data_stored":939056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574913,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574914,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574913,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":761856,"data_stored":349153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574911,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574912,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27772,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344068,"statfs":{"total":96636764160,"available":96608325632,"internally_reserved":0,"allocated":770048,"data_stored":355049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":765952,"data_stored":348731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574913,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28356,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1363968,"data_stored":945471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:06:59.813 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T14:06:59.921 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.921+0000 7fbae2f34640 1 Processor -- start 2024-04-25T14:06:59.922 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.921+0000 7fbae2f34640 1 -- start start 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae2f34640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbadc0d1120 0x7fbadc070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae2f34640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fbadc161b10 0x7fbadc161f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae2f34640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc162460 con 0x7fbadc154950 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae2f34640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc0d1530 con 0x7fbadc0d1120 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae2f34640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc0aec30 con 0x7fbadc161b10 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbadc0d1120 0x7fbadc070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fbadc0d1120 0x7fbadc070ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33526/0 (socket says 172.21.15.150:33526) 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae14d1640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fbadc154950 0x7fbadc154d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46412/0 (socket says 172.21.15.150:46412) 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 -- 172.21.15.150:0/2012052496 learned_addr learned my addr 172.21.15.150:0/2012052496 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.924 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbadbfff640 1 --2- 172.21.15.150:0/2012052496 >> v2:172.21.15.187:3300/0 conn(0x7fbadc161b10 0x7fbadc161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 -- 172.21.15.150:0/2012052496 >> v2:172.21.15.187:3300/0 conn(0x7fbadc161b10 msgr2=0x7fbadc161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/2012052496 >> v2:172.21.15.187:3300/0 conn(0x7fbadc161b10 0x7fbadc161f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 -- 172.21.15.150:0/2012052496 >> v1:172.21.15.150:6789/0 conn(0x7fbadc154950 legacy=0x7fbadc154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbadb7fe640 1 -- 172.21.15.150:0/2012052496 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2047655963 0 0) 0x7fbad4001490 con 0x7fbadc154950 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.923+0000 7fbae0cd0640 1 -- 172.21.15.150:0/2012052496 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc0aef10 con 0x7fbadc0d1120 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.924+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/2012052496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc0d1120 0x7fbadc070ef0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0b1260 tx=0x7fbacc02a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=237d1e5cc8b4c3f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.924+0000 7fbadb7fe640 1 -- 172.21.15.150:0/2012052496 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc037070 con 0x7fbadc0d1120 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.924+0000 7fbadb7fe640 1 -- 172.21.15.150:0/2012052496 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbacc032440 con 0x7fbadc0d1120 2024-04-25T14:06:59.925 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.924+0000 7fbadb7fe640 1 -- 172.21.15.150:0/2012052496 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc032810 con 0x7fbadc0d1120 2024-04-25T14:06:59.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 -- 172.21.15.150:0/2012052496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc0d1120 msgr2=0x7fbadc070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 --2- 172.21.15.150:0/2012052496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc0d1120 0x7fbadc070ef0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fbadc0b1260 tx=0x7fbacc02a150 comp rx=0 tx=0).stop 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 -- 172.21.15.150:0/2012052496 shutdown_connections 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 --2- 172.21.15.150:0/2012052496 >> v2:172.21.15.187:3300/0 conn(0x7fbadc161b10 0x7fbadc161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 --2- 172.21.15.150:0/2012052496 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc0d1120 0x7fbadc070ef0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 -- 172.21.15.150:0/2012052496 >> 172.21.15.150:0/2012052496 conn(0x7fbadc0d1ae0 msgr2=0x7fbadc0d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 -- 172.21.15.150:0/2012052496 shutdown_connections 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.925+0000 7fbae2f34640 1 -- 172.21.15.150:0/2012052496 wait complete. 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.926+0000 7fbae2f34640 1 Processor -- start 2024-04-25T14:06:59.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.926+0000 7fbae2f34640 1 -- start start 2024-04-25T14:06:59.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 0x7fbadc154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 0x7fbadc160720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc071430 con 0x7fbadc154950 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0af4d0 con 0x7fbadc161b10 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae2f34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc0715b0 con 0x7fbadc0d1120 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33536/0 (socket says 172.21.15.150:33536) 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 -- 172.21.15.150:0/433238205 learned_addr learned my addr 172.21.15.150:0/433238205 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae14d1640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 0x7fbadc160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 0x7fbadc154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 -- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 msgr2=0x7fbadc154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 0x7fbadc154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 -- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 msgr2=0x7fbadc160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 0x7fbadc160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 -- 172.21.15.150:0/433238205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc1e79d0 con 0x7fbadc154950 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 0x7fbadc154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbae14d1640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 0x7fbadc160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.928+0000 7fbadbfff640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbad000d5f0 tx=0x7fbad000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.929+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad00125f0 con 0x7fbadc154950 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.929+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbad0019070 con 0x7fbadc154950 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.929+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc1e7cb0 con 0x7fbadc154950 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.929+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad00129d0 con 0x7fbadc154950 2024-04-25T14:06:59.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.929+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc1e8270 con 0x7fbadc154950 2024-04-25T14:06:59.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.930+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbadc157e80 con 0x7fbadc154950 2024-04-25T14:06:59.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.930+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fbad0012b70 con 0x7fbadc154950 2024-04-25T14:06:59.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.931+0000 7fbad97fa640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 0x7fbac80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:06:59.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.931+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 0x7fbac80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:06:59.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.931+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fbad00954c0 con 0x7fbadc154950 2024-04-25T14:06:59.933 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.931+0000 7fbae0cd0640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 0x7fbac80769c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbacc034000 tx=0x7fbacc02e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:06:59.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:06:59.935+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbad005fe00 con 0x7fbadc154950 2024-04-25T14:07:00.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.117+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbadc076a40 con 0x7fbadc154950 2024-04-25T14:07:00.118 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.117+0000 7fbad97fa640 1 -- 172.21.15.150:0/433238205 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v65) ==== 74+0+13423 (secure 0 0 0) 0x7fbad005f700 con 0x7fbadc154950 2024-04-25T14:07:00.118 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:07:00.118 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":65,"fsid":"77163e42-55ae-4cab-b506-bd8c531cc087","created":"2024-04-25T14:01:44.407975+0000","modified":"2024-04-25T14:06:57.020126+0000","last_up_change":"2024-04-25T14:01:55.708240+0000","last_in_change":"2024-04-25T14:01:46.640223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T14:01:56.603175+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8fcbee4e-704e-4ce9-9c0a-8cebc3ea65c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6817","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6820","nonce":960179440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6822","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6825","nonce":960179440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6830","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6831","nonce":960179440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6827","nonce":960179440},{"type":"v1","addr":"172.21.15.150:6829","nonce":960179440}]},"public_addr":"172.21.15.150:6820/960179440","cluster_addr":"172.21.15.150:6825/960179440","heartbeat_back_addr":"172.21.15.150:6831/960179440","heartbeat_front_addr":"172.21.15.150:6829/960179440","state":["exists","up"]},{"osd":1,"uuid":"377acd06-0a97-46c7-9f7e-904629dd9b46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6816","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6818","nonce":715146204}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6819","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6821","nonce":715146204}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6826","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6828","nonce":715146204}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6823","nonce":715146204},{"type":"v1","addr":"172.21.15.150:6824","nonce":715146204}]},"public_addr":"172.21.15.150:6818/715146204","cluster_addr":"172.21.15.150:6821/715146204","heartbeat_back_addr":"172.21.15.150:6828/715146204","heartbeat_front_addr":"172.21.15.150:6824/715146204","state":["exists","up"]},{"osd":2,"uuid":"09bfcf89-4e73-43f8-9276-41f57000458f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6808","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6809","nonce":2809582321}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6810","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6811","nonce":2809582321}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6814","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6815","nonce":2809582321}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6812","nonce":2809582321},{"type":"v1","addr":"172.21.15.150:6813","nonce":2809582321}]},"public_addr":"172.21.15.150:6809/2809582321","cluster_addr":"172.21.15.150:6811/2809582321","heartbeat_back_addr":"172.21.15.150:6815/2809582321","heartbeat_front_addr":"172.21.15.150:6813/2809582321","state":["exists","up"]},{"osd":3,"uuid":"34d07c8c-b115-4f61-a34d-e6133022dc1e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6800","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6801","nonce":690753092}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6802","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6803","nonce":690753092}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6806","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6807","nonce":690753092}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6804","nonce":690753092},{"type":"v1","addr":"172.21.15.150:6805","nonce":690753092}]},"public_addr":"172.21.15.150:6801/690753092","cluster_addr":"172.21.15.150:6803/690753092","heartbeat_back_addr":"172.21.15.150:6807/690753092","heartbeat_front_addr":"172.21.15.150:6805/690753092","state":["exists","up"]},{"osd":4,"uuid":"17dd1c65-44ee-44c4-9f4d-00a5884fb4dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6824","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6825","nonce":361202506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6826","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6827","nonce":361202506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6830","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6831","nonce":361202506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6828","nonce":361202506},{"type":"v1","addr":"172.21.15.187:6829","nonce":361202506}]},"public_addr":"172.21.15.187:6825/361202506","cluster_addr":"172.21.15.187:6827/361202506","heartbeat_back_addr":"172.21.15.187:6831/361202506","heartbeat_front_addr":"172.21.15.187:6829/361202506","state":["exists","up"]},{"osd":5,"uuid":"19c9fb99-f307-4eab-99bf-1c6cbafc1148","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6800","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6801","nonce":3786766423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6802","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6803","nonce":3786766423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6806","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6807","nonce":3786766423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6804","nonce":3786766423},{"type":"v1","addr":"172.21.15.187:6805","nonce":3786766423}]},"public_addr":"172.21.15.187:6801/3786766423","cluster_addr":"172.21.15.187:6803/3786766423","heartbeat_back_addr":"172.21.15.187:6807/3786766423","heartbeat_front_addr":"172.21.15.187:6805/3786766423","state":["exists","up"]},{"osd":6,"uuid":"c125fda9-d826-4710-a37d-d5d2f8b67818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6808","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6809","nonce":3350780520}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6810","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6811","nonce":3350780520}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6814","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6815","nonce":3350780520}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6812","nonce":3350780520},{"type":"v1","addr":"172.21.15.187:6813","nonce":3350780520}]},"public_addr":"172.21.15.187:6809/3350780520","cluster_addr":"172.21.15.187:6811/3350780520","heartbeat_back_addr":"172.21.15.187:6815/3350780520","heartbeat_front_addr":"172.21.15.187:6813/3350780520","state":["exists","up"]},{"osd":7,"uuid":"98942485-6160-4b36-8909-62f908571405","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6816","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6817","nonce":931266284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6818","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6819","nonce":931266284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6822","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6823","nonce":931266284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6820","nonce":931266284},{"type":"v1","addr":"172.21.15.187:6821","nonce":931266284}]},"public_addr":"172.21.15.187:6817/931266284","cluster_addr":"172.21.15.187:6819/931266284","heartbeat_back_addr":"172.21.15.187:6823/931266284","heartbeat_front_addr":"172.21.15.187:6821/931266284","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.794006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.849687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.802527+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.722779+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.831883+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.804415+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.815238+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T14:01:51.824016+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.150:6836/685573768":"2024-04-26T14:06:54.007874+0000","172.21.15.150:6834/685573768":"2024-04-26T14:06:54.007874+0000","172.21.15.187:6835/133937444":"2024-04-26T14:04:37.253920+0000","172.21.15.187:6833/133937444":"2024-04-26T14:04:37.253920+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T14:07:00.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 msgr2=0x7fbac80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:00.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 0x7fbac80769c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fbacc034000 tx=0x7fbacc02e000 comp rx=0 tx=0).stop 2024-04-25T14:07:00.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 msgr2=0x7fbadc1573e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:00.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbad000d5f0 tx=0x7fbad000dab0 comp rx=0 tx=0).stop 2024-04-25T14:07:00.120 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 shutdown_connections 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fbac80744b0 0x7fbac80769c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fbadc161b10 0x7fbadc160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fbadc154950 0x7fbadc1573e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 --2- 172.21.15.150:0/433238205 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fbadc0d1120 0x7fbadc154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 >> 172.21.15.150:0/433238205 conn(0x7fbadc0d1ae0 msgr2=0x7fbadc15c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 shutdown_connections 2024-04-25T14:07:00.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:00.120+0000 7fbae2f34640 1 -- 172.21.15.150:0/433238205 wait complete. 2024-04-25T14:07:01.132 INFO:tasks.ceph:Scrubbing osd.0 2024-04-25T14:07:01.133 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:01.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.247+0000 7f70022b1640 1 Processor -- start 2024-04-25T14:07:01.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.247+0000 7f70022b1640 1 -- start start 2024-04-25T14:07:01.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f70022b1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0d1210 0x7f6ffc15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f70022b1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ffc161ae0 con 0x7f6ffc15f900 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f70022b1640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ffc161dc0 con 0x7f6ffc0739c0 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f70022b1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f6ffc0d1620 con 0x7f6ffc0d1210 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0d1210 0x7f6ffc15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f6ffc0d1210 0x7f6ffc15ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33550/0 (socket says 172.21.15.150:33550) 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 -- 172.21.15.150:0/116568989 learned_addr learned my addr 172.21.15.150:0/116568989 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 -- 172.21.15.150:0/116568989 >> v1:172.21.15.187:6789/0 conn(0x7f6ffc0739c0 legacy=0x7f6ffc073dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 -- 172.21.15.150:0/116568989 >> v1:172.21.15.150:6789/0 conn(0x7f6ffc15f900 legacy=0x7f6ffc15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffaffd640 1 -- 172.21.15.150:0/116568989 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3369041878 0 0) 0x7f6ffc161ae0 con 0x7f6ffc15f900 2024-04-25T14:07:01.251 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.250+0000 7f6ffbfff640 1 -- 172.21.15.150:0/116568989 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc1620a0 con 0x7f6ffc0d1210 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.251+0000 7f6ffbfff640 1 --2- 172.21.15.150:0/116568989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc15ef90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d28a68f9afdf051 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.251+0000 7f6ffaffd640 1 -- 172.21.15.150:0/116568989 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec037070 con 0x7f6ffc0d1210 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.251+0000 7f6ffaffd640 1 -- 172.21.15.150:0/116568989 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fec02ae50 con 0x7f6ffc0d1210 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.251+0000 7f6ffaffd640 1 -- 172.21.15.150:0/116568989 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec032860 con 0x7f6ffc0d1210 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 -- 172.21.15.150:0/116568989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0d1210 msgr2=0x7f6ffc15ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 --2- 172.21.15.150:0/116568989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc15ef90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec02a300 comp rx=0 tx=0).stop 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 -- 172.21.15.150:0/116568989 shutdown_connections 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 --2- 172.21.15.150:0/116568989 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc15ef90 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 -- 172.21.15.150:0/116568989 >> 172.21.15.150:0/116568989 conn(0x7f6ffc0d1f80 msgr2=0x7f6ffc0732b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 -- 172.21.15.150:0/116568989 shutdown_connections 2024-04-25T14:07:01.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.252+0000 7f70022b1640 1 -- 172.21.15.150:0/116568989 wait complete. 2024-04-25T14:07:01.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.254+0000 7f70022b1640 1 Processor -- start 2024-04-25T14:07:01.254 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.254+0000 7f70022b1640 1 -- start start 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 0x7f6ffc159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ffc15f900 0x7f6ffc1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc162940 con 0x7f6ffc0739c0 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f6ffc162660 con 0x7f6ffc0d1210 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f70022b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f6ffc0afb50 con 0x7f6ffc15f900 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f6ffbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 0x7f6ffc159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f6ffbfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 0x7f6ffc159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33564/0 (socket says 172.21.15.150:33564) 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f6ffbfff640 1 -- 172.21.15.150:0/4101236725 learned_addr learned my addr 172.21.15.150:0/4101236725 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f6ffb7fe640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.255+0000 7f700084e640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ffc15f900 0x7f6ffc1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 -- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ffc15f900 msgr2=0x7f6ffc1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ffc15f900 0x7f6ffc1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.257 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 -- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ffc0d1210 msgr2=0x7f6ffc157fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f6ffc0d1210 0x7f6ffc157fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ffc1e8990 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f700084e640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f6ffc15f900 0x7f6ffc1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ffbfff640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 0x7f6ffc159140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec041070 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fec03c870 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ffc1e8c70 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fec037040 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.256+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ffc1e9230 con 0x7f6ffc0739c0 2024-04-25T14:07:01.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.257+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ffc076a40 con 0x7f6ffc0739c0 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.258+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f6fec044070 con 0x7f6ffc0739c0 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.258+0000 7f6ff8ff9640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6fdc0744e0 0x7f6fdc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f6fec0b5f90 con 0x7f6ffc0739c0 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ffb7fe640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6fdc0744e0 0x7f6fdc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ff8ff9640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7f6fdc07aa70 0x7f6fdc07cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6fec00af30 con 0x7f6fdc07aa70 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f6fec0b6420 con 0x7f6ffc0739c0 2024-04-25T14:07:01.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f700084e640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7f6fdc07aa70 0x7f6fdc07cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f6ffb7fe640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6fdc0744e0 0x7f6fdc0769f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0d0bc0 tx=0x7f6fe4009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.259+0000 7f700084e640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7f6fdc07aa70 0x7f6fdc07cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.260+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== osd.0 v2:172.21.15.150:6817/960179440 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6fec00af30 con 0x7f6fdc07aa70 2024-04-25T14:07:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 --> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f6ffc06d610 con 0x7f6fdc07aa70 2024-04-25T14:07:01.273 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f6ff8ff9640 1 -- 172.21.15.150:0/4101236725 <== osd.0 v2:172.21.15.150:6817/960179440 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f6ffc06d610 con 0x7f6fdc07aa70 2024-04-25T14:07:01.273 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:01.273 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:01.273 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7f6fdc07aa70 msgr2=0x7f6fdc07cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6817/960179440,v1:172.21.15.150:6820/960179440] conn(0x7f6fdc07aa70 0x7f6fdc07cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6fdc0744e0 msgr2=0x7f6fdc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f6fdc0744e0 0x7f6fdc0769f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0d0bc0 tx=0x7f6fe4009040 comp rx=0 tx=0).stop 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 msgr2=0x7f6ffc159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.274 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 --2- 172.21.15.150:0/4101236725 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f6ffc0739c0 0x7f6ffc159140 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f6fec009870 tx=0x7f6fec0046c0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f700084e640 1 -- 172.21.15.150:0/4101236725 reap_dead start 2024-04-25T14:07:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 shutdown_connections 2024-04-25T14:07:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.273+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 >> 172.21.15.150:0/4101236725 conn(0x7f6ffc0d1f80 msgr2=0x7f6ffc0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.274+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 shutdown_connections 2024-04-25T14:07:01.275 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.274+0000 7f70022b1640 1 -- 172.21.15.150:0/4101236725 wait complete. 2024-04-25T14:07:01.286 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-25T14:07:01.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.396+0000 7f41ac384640 1 Processor -- start 2024-04-25T14:07:01.396 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.396+0000 7f41ac384640 1 -- start start 2024-04-25T14:07:01.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41ac384640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41a40735b0 0x7f41a4160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.398 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41ac384640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f41a40cf330 0x7f41a40ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41ac384640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a4164e10 con 0x7f41a40d3290 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41ac384640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f41a4160770 con 0x7f41a40735b0 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41ac384640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f41a40b6c70 con 0x7f41a40cf330 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41a40735b0 0x7f41a4160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f41a40735b0 0x7f41a4160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33570/0 (socket says 172.21.15.150:33570) 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 -- 172.21.15.150:0/2753965225 learned_addr learned my addr 172.21.15.150:0/2753965225 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41aa921640 1 --2- 172.21.15.150:0/2753965225 >> v2:172.21.15.187:3300/0 conn(0x7f41a40cf330 0x7f41a40ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 -- 172.21.15.150:0/2753965225 >> v2:172.21.15.187:3300/0 conn(0x7f41a40cf330 msgr2=0x7f41a40ce000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 --2- 172.21.15.150:0/2753965225 >> v2:172.21.15.187:3300/0 conn(0x7f41a40cf330 0x7f41a40ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 -- 172.21.15.150:0/2753965225 >> v1:172.21.15.187:6789/0 conn(0x7f41a40d3290 legacy=0x7f41a4072ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.399 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.398+0000 7f41a991f640 1 -- 172.21.15.150:0/2753965225 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a4164050 con 0x7f41a40735b0 2024-04-25T14:07:01.400 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.399+0000 7f41a991f640 1 --2- 172.21.15.150:0/2753965225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40735b0 0x7f41a4160230 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f41a40d3980 tx=0x7f419402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4940edf9668f174 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.399+0000 7f41a911e640 1 -- 172.21.15.150:0/2753965225 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194037070 con 0x7f41a40735b0 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.399+0000 7f41a911e640 1 -- 172.21.15.150:0/2753965225 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4194032440 con 0x7f41a40735b0 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.399+0000 7f41a911e640 1 -- 172.21.15.150:0/2753965225 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194032810 con 0x7f41a40735b0 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 -- 172.21.15.150:0/2753965225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40735b0 msgr2=0x7f41a4160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 --2- 172.21.15.150:0/2753965225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40735b0 0x7f41a4160230 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f41a40d3980 tx=0x7f419402a1a0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 -- 172.21.15.150:0/2753965225 shutdown_connections 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 --2- 172.21.15.150:0/2753965225 >> v2:172.21.15.187:3300/0 conn(0x7f41a40cf330 0x7f41a40ce000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 --2- 172.21.15.150:0/2753965225 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40735b0 0x7f41a4160230 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 -- 172.21.15.150:0/2753965225 >> 172.21.15.150:0/2753965225 conn(0x7f41a40d39b0 msgr2=0x7f41a40d3dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 -- 172.21.15.150:0/2753965225 shutdown_connections 2024-04-25T14:07:01.401 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.400+0000 7f41ac384640 1 -- 172.21.15.150:0/2753965225 wait complete. 2024-04-25T14:07:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.401+0000 7f41ac384640 1 Processor -- start 2024-04-25T14:07:01.402 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.402+0000 7f41ac384640 1 -- start start 2024-04-25T14:07:01.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a4070af0 0x7f41a40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41a4159e50 0x7f41a415a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.404 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41648f0 con 0x7f41a40cf330 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f41a4164610 con 0x7f41a4070af0 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41ac384640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f41a40b6800 con 0x7f41a4159e50 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33578/0 (socket says 172.21.15.150:33578) 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 -- 172.21.15.150:0/1896713842 learned_addr learned my addr 172.21.15.150:0/1896713842 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa921640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41a4159e50 0x7f41a415a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41a991f640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a4070af0 0x7f41a40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 -- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41a4159e50 msgr2=0x7f41a415a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41a4159e50 0x7f41a415a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 -- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a4070af0 msgr2=0x7f41a40b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a4070af0 0x7f41a40b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 -- 172.21.15.150:0/1896713842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a415a7a0 con 0x7f41a40cf330 2024-04-25T14:07:01.405 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.403+0000 7f41aa120640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f41a000ba50 tx=0x7f41a000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.404+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0019070 con 0x7f41a40cf330 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.404+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41a000cc20 con 0x7f41a40cf330 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.404+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a406dfb0 con 0x7f41a40cf330 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.404+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0014680 con 0x7f41a40cf330 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.404+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a406e570 con 0x7f41a40cf330 2024-04-25T14:07:01.406 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.405+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a4076a40 con 0x7f41a40cf330 2024-04-25T14:07:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.405+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f41a001c070 con 0x7f41a40cf330 2024-04-25T14:07:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.406+0000 7f419affd640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 0x7f41880769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.406+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f41a0094d70 con 0x7f41a40cf330 2024-04-25T14:07:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.406+0000 7f41a991f640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 0x7f41880769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.407 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.406+0000 7f41a991f640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 0x7f41880769c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4194034000 tx=0x7f419402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.410 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.411+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41a415a7a0 con 0x7f41a40cf330 2024-04-25T14:07:01.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.585+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f41a40d3000 con 0x7f41880744b0 2024-04-25T14:07:01.585 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.585+0000 7f419affd640 1 -- 172.21.15.150:0/1896713842 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f41a40d3000 con 0x7f41880744b0 2024-04-25T14:07:01.585 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 0 to deep-scrub 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 msgr2=0x7f41880769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 0x7f41880769c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4194034000 tx=0x7f419402e000 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 msgr2=0x7f41a40b3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f41a000ba50 tx=0x7f41a000bf10 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 shutdown_connections 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f41880744b0 0x7f41880769c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f41a4159e50 0x7f41a415a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f41a4070af0 0x7f41a40b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 --2- 172.21.15.150:0/1896713842 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f41a40cf330 0x7f41a40b3ac0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 >> 172.21.15.150:0/1896713842 conn(0x7f41a40d39b0 msgr2=0x7f41a40d22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.588 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 shutdown_connections 2024-04-25T14:07:01.589 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.587+0000 7f41ac384640 1 -- 172.21.15.150:0/1896713842 wait complete. 2024-04-25T14:07:01.599 INFO:tasks.ceph:Scrubbing osd.1 2024-04-25T14:07:01.599 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:01.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.711+0000 7efd28096640 1 Processor -- start 2024-04-25T14:07:01.712 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.711+0000 7efd28096640 1 -- start start 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd28096640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7efd200b5b30 0x7efd200b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd28096640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd200d3240 con 0x7efd20161be0 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd28096640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd20162880 con 0x7efd200d2e30 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd28096640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7efd200b3260 con 0x7efd200b5b30 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd26633640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7efd20161be0 0x7efd20161ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46422/0 (socket says 172.21.15.150:46422) 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd26633640 1 -- 172.21.15.150:0/3114980482 learned_addr learned my addr 172.21.15.150:0/3114980482 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 989796543 0 0) 0x7efd200d3240 con 0x7efd20161be0 2024-04-25T14:07:01.715 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd0c003520 con 0x7efd20161be0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd25631640 1 --2- 172.21.15.150:0/3114980482 >> v2:172.21.15.187:3300/0 conn(0x7efd200b5b30 0x7efd200b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 30150082 0 0) 0x7efd0c003520 con 0x7efd20161be0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 >> v2:172.21.15.187:3300/0 conn(0x7efd200b5b30 msgr2=0x7efd200b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 --2- 172.21.15.150:0/3114980482 >> v2:172.21.15.187:3300/0 conn(0x7efd200b5b30 0x7efd200b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 >> v1:172.21.15.187:6789/0 conn(0x7efd200d2e30 legacy=0x7efd200b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd200d2350 con 0x7efd20161be0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7efd18001670 con 0x7efd20161be0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.713+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 >> v1:172.21.15.150:6789/0 conn(0x7efd20161be0 legacy=0x7efd20161ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd0c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd0c004800 0x7efd0c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd0c005270 0x7efd0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd200b3260 con 0x7efd0c0039b0 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efd0c00c1f0 con 0x7efd0c004800 2024-04-25T14:07:01.716 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd0c00c3f0 con 0x7efd0c005270 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25631640 1 --2- 172.21.15.150:0/3114980482 >> v2:172.21.15.187:3300/0 conn(0x7efd200b5b30 0x7efd200b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd0c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd26633640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd0c005270 0x7efd0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25631640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd0c004800 0x7efd0c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 -- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd0c005270 msgr2=0x7efd0c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd0c005270 0x7efd0c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 -- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd0c004800 msgr2=0x7efd0c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd0c004800 0x7efd0c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd26633640 1 -- 172.21.15.150:0/3114980482 reap_dead start 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25e32640 1 -- 172.21.15.150:0/3114980482 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0c003520 con 0x7efd0c0039b0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.714+0000 7efd25631640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd0c004800 0x7efd0c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.715+0000 7efd25e32640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd0c003e20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001e50 tx=0x7efd0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dcfac07e33c0c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.715+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c001a70 con 0x7efd0c0039b0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.715+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c02ae40 con 0x7efd0c0039b0 2024-04-25T14:07:01.717 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.715+0000 7efd24e30640 1 -- 172.21.15.150:0/3114980482 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c003bb0 con 0x7efd0c0039b0 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 -- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 msgr2=0x7efd0c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd0c003e20 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efd1c001e50 tx=0x7efd0c005830 comp rx=0 tx=0).stop 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 -- 172.21.15.150:0/3114980482 shutdown_connections 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 --2- 172.21.15.150:0/3114980482 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd0c003e20 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 -- 172.21.15.150:0/3114980482 >> 172.21.15.150:0/3114980482 conn(0x7efd200d0420 msgr2=0x7efd200d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 -- 172.21.15.150:0/3114980482 shutdown_connections 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.716+0000 7efd28096640 1 -- 172.21.15.150:0/3114980482 wait complete. 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.717+0000 7efd28096640 1 Processor -- start 2024-04-25T14:07:01.720 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.717+0000 7efd28096640 1 -- start start 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd20160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd20161110 0x7efd200b5350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd2015f040 0x7efd200b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7efd200b2d20 con 0x7efd0c0039b0 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7efd200d29a0 con 0x7efd20161110 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7efd200b2ea0 con 0x7efd2015f040 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd20160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd20160bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33590/0 (socket says 172.21.15.150:33590) 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 -- 172.21.15.150:0/952650914 learned_addr learned my addr 172.21.15.150:0/952650914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 -- 172.21.15.150:0/952650914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd2015f040 msgr2=0x7efd200b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7efd2015f040 0x7efd200b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 -- 172.21.15.150:0/952650914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd20161110 msgr2=0x7efd200b5350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7efd20161110 0x7efd200b5350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.721 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd201615e0 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd25e32640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd20160bd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7efd1c005040 tx=0x7efd1c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c02ac40 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c0050d0 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2006dff0 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.719+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c0054a0 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.720+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2006e5b0 con 0x7efd0c0039b0 2024-04-25T14:07:01.722 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.720+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efd20076a40 con 0x7efd0c0039b0 2024-04-25T14:07:01.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.721+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7efd1c03a070 con 0x7efd0c0039b0 2024-04-25T14:07:01.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd167fc640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efd040744b0 0x7efd040769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd25631640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efd040744b0 0x7efd040769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.723 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7efd1c0b6030 con 0x7efd0c0039b0 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd167fc640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7efd0407aa40 0x7efd0407ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd1c00bf30 con 0x7efd0407aa40 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7efd1c0b6510 con 0x7efd0c0039b0 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd26633640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7efd0407aa40 0x7efd0407ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd25631640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efd040744b0 0x7efd040769c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efd10006f00 tx=0x7efd10006df0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.722+0000 7efd26633640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7efd0407aa40 0x7efd0407ced0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.724 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.723+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== osd.1 v2:172.21.15.150:6816/715146204 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efd1c00bf30 con 0x7efd0407aa40 2024-04-25T14:07:01.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.736+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 --> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efd200cdb30 con 0x7efd0407aa40 2024-04-25T14:07:01.736 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.736+0000 7efd167fc640 1 -- 172.21.15.150:0/952650914 <== osd.1 v2:172.21.15.150:6816/715146204 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efd200cdb30 con 0x7efd0407aa40 2024-04-25T14:07:01.736 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:01.736 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:01.737 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:01.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.736+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7efd0407aa40 msgr2=0x7efd0407ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.737 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.736+0000 7efd28096640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6816/715146204,v1:172.21.15.150:6818/715146204] conn(0x7efd0407aa40 0x7efd0407ced0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efd040744b0 msgr2=0x7efd040769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7efd040744b0 0x7efd040769c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7efd10006f00 tx=0x7efd10006df0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 msgr2=0x7efd20160bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 --2- 172.21.15.150:0/952650914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7efd0c0039b0 0x7efd20160bd0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7efd1c005040 tx=0x7efd1c0014a0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd26633640 1 -- 172.21.15.150:0/952650914 reap_dead start 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 shutdown_connections 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 >> 172.21.15.150:0/952650914 conn(0x7efd200d0420 msgr2=0x7efd20072cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 shutdown_connections 2024-04-25T14:07:01.738 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.737+0000 7efd28096640 1 -- 172.21.15.150:0/952650914 wait complete. 2024-04-25T14:07:01.747 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-25T14:07:01.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.856+0000 7fb3b2c34640 1 Processor -- start 2024-04-25T14:07:01.857 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.857+0000 7fb3b2c34640 1 -- start start 2024-04-25T14:07:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 0x7fb3ac06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3ac163d80 0x7fb3ac0b53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.860 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 0x7fb3ac072f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ac06ff50 con 0x7fb3ac163d80 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb3ac0b5d10 con 0x7fb3ac0b5900 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b2c34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ac06f3d0 con 0x7fb3ac160460 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3ac163d80 0x7fb3ac0b53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb3ac163d80 0x7fb3ac0b53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33600/0 (socket says 172.21.15.150:33600) 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 -- 172.21.15.150:0/1666673180 learned_addr learned my addr 172.21.15.150:0/1666673180 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b09d0640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 0x7fb3ac06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b11d1640 1 --2- 172.21.15.150:0/1666673180 >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 0x7fb3ac072f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 -- 172.21.15.150:0/1666673180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 msgr2=0x7fb3ac06ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 0x7fb3ac06ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 -- 172.21.15.150:0/1666673180 >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 msgr2=0x7fb3ac072f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1666673180 >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 0x7fb3ac072f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.861 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3abfff640 1 -- 172.21.15.150:0/1666673180 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac06f6b0 con 0x7fb3ac163d80 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b11d1640 1 --2- 172.21.15.150:0/1666673180 >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 0x7fb3ac072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.858+0000 7fb3b09d0640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 0x7fb3ac06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.859+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac0b53c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac0743d0 tx=0x7fb3a002a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5e3db4ecd3e142 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.859+0000 7fb3ab7fe640 1 -- 172.21.15.150:0/1666673180 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0037070 con 0x7fb3ac163d80 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.859+0000 7fb3ab7fe640 1 -- 172.21.15.150:0/1666673180 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a0032440 con 0x7fb3ac163d80 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.859+0000 7fb3ab7fe640 1 -- 172.21.15.150:0/1666673180 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0032810 con 0x7fb3ac163d80 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1666673180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 msgr2=0x7fb3ac0b53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac0b53c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac0743d0 tx=0x7fb3a002a140 comp rx=0 tx=0).stop 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1666673180 shutdown_connections 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1666673180 >> v2:172.21.15.187:3300/0 conn(0x7fb3ac0b5900 0x7fb3ac072f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac0b53c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1666673180 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac160460 0x7fb3ac06ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1666673180 >> 172.21.15.150:0/1666673180 conn(0x7fb3ac0d03c0 msgr2=0x7fb3ac0d07d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1666673180 shutdown_connections 2024-04-25T14:07:01.862 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.860+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1666673180 wait complete. 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.862+0000 7fb3b2c34640 1 Processor -- start 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.862+0000 7fb3b2c34640 1 -- start start 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3ac0b5900 0x7fb3ac0caac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac0cd820 0x7fb3ac15e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac0b3030 con 0x7fb3ac163d80 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb3ac06fc70 con 0x7fb3ac0b5900 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3b2c34640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb3ac0ce5c0 con 0x7fb3ac0cd820 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.865 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33604/0 (socket says 172.21.15.150:33604) 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.863+0000 7fb3abfff640 1 -- 172.21.15.150:0/1361695063 learned_addr learned my addr 172.21.15.150:0/1361695063 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3b11d1640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac0cd820 0x7fb3ac15e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3b09d0640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3ac0b5900 0x7fb3ac0caac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 -- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac0cd820 msgr2=0x7fb3ac15e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac0cd820 0x7fb3ac15e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 -- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3ac0b5900 msgr2=0x7fb3ac0caac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3ac0b5900 0x7fb3ac0caac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 -- 172.21.15.150:0/1361695063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3ac0b70f0 con 0x7fb3ac163d80 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3abfff640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002a180 tx=0x7fb3a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0041020 con 0x7fb3ac163d80 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a00306c0 con 0x7fb3ac163d80 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3ac0b73d0 con 0x7fb3ac163d80 2024-04-25T14:07:01.866 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0037040 con 0x7fb3ac163d80 2024-04-25T14:07:01.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.864+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3ac0b7990 con 0x7fb3ac163d80 2024-04-25T14:07:01.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.865+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3ac076a40 con 0x7fb3ac163d80 2024-04-25T14:07:01.867 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.866+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb3a0044020 con 0x7fb3ac163d80 2024-04-25T14:07:01.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.866+0000 7fb3a97fa640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 0x7fb3980769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:01.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.866+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fb3a00b5740 con 0x7fb3ac163d80 2024-04-25T14:07:01.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.866+0000 7fb3b09d0640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 0x7fb3980769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:01.868 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.867+0000 7fb3b09d0640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 0x7fb3980769c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb39c006fd0 tx=0x7fb39c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:01.871 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:01.872+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3a004b030 con 0x7fb3ac163d80 2024-04-25T14:07:02.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.051+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fb3ac0cb340 con 0x7fb3980744b0 2024-04-25T14:07:02.051 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.051+0000 7fb3a97fa640 1 -- 172.21.15.150:0/1361695063 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb3ac0cb340 con 0x7fb3980744b0 2024-04-25T14:07:02.051 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 1 to deep-scrub 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 msgr2=0x7fb3980769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 0x7fb3980769c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fb39c006fd0 tx=0x7fb39c008040 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 msgr2=0x7fb3ac1632d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fb3a002a180 tx=0x7fb3a00046c0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 shutdown_connections 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3980744b0 0x7fb3980769c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb3ac0cd820 0x7fb3ac15e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb3ac163d80 0x7fb3ac1632d0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 --2- 172.21.15.150:0/1361695063 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb3ac0b5900 0x7fb3ac0caac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 >> 172.21.15.150:0/1361695063 conn(0x7fb3ac0d03c0 msgr2=0x7fb3ac0d20e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.054 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 shutdown_connections 2024-04-25T14:07:02.055 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.053+0000 7fb3b2c34640 1 -- 172.21.15.150:0/1361695063 wait complete. 2024-04-25T14:07:02.064 INFO:tasks.ceph:Scrubbing osd.2 2024-04-25T14:07:02.064 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:02.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.175+0000 7f14bbfb2640 1 Processor -- start 2024-04-25T14:07:02.176 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.175+0000 7f14bbfb2640 1 -- start start 2024-04-25T14:07:02.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14bbfb2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b406ec00 0x7f14b40d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14bbfb2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b40d1800 0x7f14b40b5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14bbfb2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14b40b5d70 con 0x7f14b40b5960 2024-04-25T14:07:02.178 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14bbfb2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f14b41607b0 con 0x7f14b406ec00 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14bbfb2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14b4160a10 con 0x7f14b40d1800 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b406ec00 0x7f14b40d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f14b406ec00 0x7f14b40d2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33614/0 (socket says 172.21.15.150:33614) 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14ba54f640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f14b40b5960 0x7f14b4163eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46424/0 (socket says 172.21.15.150:46424) 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 -- 172.21.15.150:0/2036201914 learned_addr learned my addr 172.21.15.150:0/2036201914 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b954d640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b40d1800 0x7f14b40b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 -- 172.21.15.150:0/2036201914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b40d1800 msgr2=0x7f14b40b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b40d1800 0x7f14b40b5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 -- 172.21.15.150:0/2036201914 >> v1:172.21.15.150:6789/0 conn(0x7f14b40b5960 legacy=0x7f14b4163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b8d4c640 1 -- 172.21.15.150:0/2036201914 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701826405 0 0) 0x7f14ac001110 con 0x7f14b40b5960 2024-04-25T14:07:02.179 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 -- 172.21.15.150:0/2036201914 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b4160cb0 con 0x7f14b406ec00 2024-04-25T14:07:02.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b9d4e640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b406ec00 0x7f14b40d2710 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009e50 tx=0x7f14b002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a050e7483dbf6472 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b8d4c640 1 -- 172.21.15.150:0/2036201914 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b0036070 con 0x7f14b406ec00 2024-04-25T14:07:02.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b8d4c640 1 -- 172.21.15.150:0/2036201914 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14b0031600 con 0x7f14b406ec00 2024-04-25T14:07:02.180 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.178+0000 7f14b8d4c640 1 -- 172.21.15.150:0/2036201914 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14b00319d0 con 0x7f14b406ec00 2024-04-25T14:07:02.181 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 -- 172.21.15.150:0/2036201914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b406ec00 msgr2=0x7f14b40d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b406ec00 0x7f14b40d2710 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f14b0009e50 tx=0x7f14b002a710 comp rx=0 tx=0).stop 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 -- 172.21.15.150:0/2036201914 shutdown_connections 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b40d1800 0x7f14b40b5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/2036201914 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b406ec00 0x7f14b40d2710 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 -- 172.21.15.150:0/2036201914 >> 172.21.15.150:0/2036201914 conn(0x7f14b40d0420 msgr2=0x7f14b40d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 -- 172.21.15.150:0/2036201914 shutdown_connections 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.179+0000 7f14bbfb2640 1 -- 172.21.15.150:0/2036201914 wait complete. 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.181+0000 7f14bbfb2640 1 Processor -- start 2024-04-25T14:07:02.182 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.181+0000 7f14bbfb2640 1 -- start start 2024-04-25T14:07:02.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b406ec00 0x7f14b40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.185 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14b40b5960 0x7f14b41f7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 0x7f14b40b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4161550 con 0x7f14b40d1800 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f14b4161270 con 0x7f14b40b5960 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14bbfb2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f14b406fd70 con 0x7f14b406ec00 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 0x7f14b40b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 0x7f14b40b8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33624/0 (socket says 172.21.15.150:33624) 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 -- 172.21.15.150:0/1034368656 learned_addr learned my addr 172.21.15.150:0/1034368656 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14b9d4e640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b406ec00 0x7f14b40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 -- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b406ec00 msgr2=0x7f14b40cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f14b406ec00 0x7f14b40cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 -- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14b40b5960 msgr2=0x7f14b41f7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14b954d640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14b40b5960 0x7f14b41f7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f14b40b5960 0x7f14b41f7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.186 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.182+0000 7f14ba54f640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b41f7770 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14ba54f640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 0x7f14b40b8960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00c410 tx=0x7f14ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac0074e0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14ac0047e0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b40ca9e0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14ac004bd0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.183+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b40cb1c0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.184+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14b41535b0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.184+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f14ac011070 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14aa7fc640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f14980744e0 0x7f14980769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14b9d4e640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f14980744e0 0x7f14980769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f14ac0954c0 con 0x7f14b40d1800 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14aa7fc640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f149807aa70 0x7f149807cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.187 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14ac01fe50 con 0x7f149807aa70 2024-04-25T14:07:02.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f14ac095950 con 0x7f14b40d1800 2024-04-25T14:07:02.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14b954d640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f149807aa70 0x7f149807cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.185+0000 7f14b9d4e640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f14980744e0 0x7f14980769f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f14b0033000 tx=0x7f14b0009000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.186+0000 7f14b954d640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f149807aa70 0x7f149807cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.188 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.186+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== osd.2 v2:172.21.15.150:6808/2809582321 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f14ac01fe50 con 0x7f149807aa70 2024-04-25T14:07:02.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.199+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 --> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f14b4076a40 con 0x7f149807aa70 2024-04-25T14:07:02.199 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.199+0000 7f14aa7fc640 1 -- 172.21.15.150:0/1034368656 <== osd.2 v2:172.21.15.150:6808/2809582321 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f14b4076a40 con 0x7f149807aa70 2024-04-25T14:07:02.200 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:02.200 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:02.200 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f149807aa70 msgr2=0x7f149807cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6808/2809582321,v1:172.21.15.150:6809/2809582321] conn(0x7f149807aa70 0x7f149807cf00 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f14980744e0 msgr2=0x7f14980769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f14980744e0 0x7f14980769f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f14b0033000 tx=0x7f14b0009000 comp rx=0 tx=0).stop 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 msgr2=0x7f14b40b8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 --2- 172.21.15.150:0/1034368656 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f14b40d1800 0x7f14b40b8960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f14ac00c410 tx=0x7f14ac00c8d0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14ba54f640 1 -- 172.21.15.150:0/1034368656 reap_dead start 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 shutdown_connections 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 >> 172.21.15.150:0/1034368656 conn(0x7f14b40d0420 msgr2=0x7f14b40b4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 shutdown_connections 2024-04-25T14:07:02.201 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.200+0000 7f14bbfb2640 1 -- 172.21.15.150:0/1034368656 wait complete. 2024-04-25T14:07:02.211 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-25T14:07:02.319 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.319+0000 7fd9918d1640 1 Processor -- start 2024-04-25T14:07:02.320 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.319+0000 7fd9918d1640 1 -- start start 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd9918d1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 0x7fd98c160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd9918d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 0x7fd98c0d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd9918d1640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd98c0b9600 con 0x7fd98c0b91f0 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd9918d1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fd98c06cc60 con 0x7fd98c0b4c10 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd9918d1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c0d4f20 con 0x7fd98c06d800 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98a7fc640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fd98c0b91f0 0x7fd98c06d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46436/0 (socket says 172.21.15.150:46436) 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98a7fc640 1 -- 172.21.15.150:0/233067089 learned_addr learned my addr 172.21.15.150:0/233067089 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/233067089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 0x7fd98c0d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98affd640 1 --2- 172.21.15.150:0/233067089 >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 0x7fd98c160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3519916939 0 0) 0x7fd98c0b9600 con 0x7fd98c0b91f0 2024-04-25T14:07:02.322 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd974003520 con 0x7fd98c0b91f0 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98b7fe640 1 -- 172.21.15.150:0/233067089 >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 msgr2=0x7fd98c160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/233067089 >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 0x7fd98c160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4276645261 0 0) 0x7fd974003520 con 0x7fd98c0b91f0 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98b7fe640 1 -- 172.21.15.150:0/233067089 >> v1:172.21.15.150:6789/0 conn(0x7fd98c0b91f0 legacy=0x7fd98c06d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98b7fe640 1 -- 172.21.15.150:0/233067089 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c0d5200 con 0x7fd98c06d800 2024-04-25T14:07:02.323 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.321+0000 7fd98affd640 1 --2- 172.21.15.150:0/233067089 >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 0x7fd98c160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T14:07:02.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.322+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/233067089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 0x7fd98c0d0950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=553c829b9a54b17a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.324 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.322+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984037070 con 0x7fd98c06d800 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.322+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98402aba0 con 0x7fd98c06d800 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.322+0000 7fd989ffb640 1 -- 172.21.15.150:0/233067089 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980001690 con 0x7fd98c06d800 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 -- 172.21.15.150:0/233067089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 msgr2=0x7fd98c0d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 --2- 172.21.15.150:0/233067089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 0x7fd98c0d0950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9840097a0 tx=0x7fd98402a110 comp rx=0 tx=0).stop 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 -- 172.21.15.150:0/233067089 shutdown_connections 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 --2- 172.21.15.150:0/233067089 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c06d800 0x7fd98c0d0950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 --2- 172.21.15.150:0/233067089 >> v2:172.21.15.187:3300/0 conn(0x7fd98c0b4c10 0x7fd98c160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 -- 172.21.15.150:0/233067089 >> 172.21.15.150:0/233067089 conn(0x7fd98c0b7690 msgr2=0x7fd98c0b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 -- 172.21.15.150:0/233067089 shutdown_connections 2024-04-25T14:07:02.325 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.323+0000 7fd9918d1640 1 -- 172.21.15.150:0/233067089 wait complete. 2024-04-25T14:07:02.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.325+0000 7fd9918d1640 1 Processor -- start 2024-04-25T14:07:02.326 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.325+0000 7fd9918d1640 1 -- start start 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c0b4c10 0x7fd98c0c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 0x7fd98c14a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c06c8f0 con 0x7fd98c06d800 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c06c610 con 0x7fd98c0b91f0 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.326+0000 7fd9918d1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fd98c06dc10 con 0x7fd98c0b4c10 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33632/0 (socket says 172.21.15.150:33632) 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 -- 172.21.15.150:0/689020706 learned_addr learned my addr 172.21.15.150:0/689020706 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98a7fc640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c0b4c10 0x7fd98c0c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 0x7fd98c14a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.328 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 -- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c0b4c10 msgr2=0x7fd98c0c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c0b4c10 0x7fd98c0c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 -- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 msgr2=0x7fd98c14a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 0x7fd98c14a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 -- 172.21.15.150:0/689020706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c15ae00 con 0x7fd98c06d800 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98b7fe640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 0x7fd98c14a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd98affd640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00efc0 tx=0x7fd97c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c018020 con 0x7fd98c06d800 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd97c014680 con 0x7fd98c06d800 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c0cfeb0 con 0x7fd98c06d800 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd97c014a50 con 0x7fd98c06d800 2024-04-25T14:07:02.329 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.327+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c0c9650 con 0x7fd98c06d800 2024-04-25T14:07:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.328+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c076a40 con 0x7fd98c06d800 2024-04-25T14:07:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.329+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fd97c005d10 con 0x7fd98c06d800 2024-04-25T14:07:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.329+0000 7fd973fff640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 0x7fd9780769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.330 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.329+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fd97c095f60 con 0x7fd98c06d800 2024-04-25T14:07:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.329+0000 7fd98a7fc640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 0x7fd9780769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.331 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.330+0000 7fd98a7fc640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 0x7fd9780769c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd98c14a8a0 tx=0x7fd980009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.334 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.335+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd97c022030 con 0x7fd98c06d800 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.511+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fd98c0d4030 con 0x7fd9780744b0 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.512+0000 7fd973fff640 1 -- 172.21.15.150:0/689020706 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fd98c0d4030 con 0x7fd9780744b0 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 2 to deep-scrub 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 msgr2=0x7fd9780769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 0x7fd9780769c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd98c14a8a0 tx=0x7fd980009040 comp rx=0 tx=0).stop 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 msgr2=0x7fd98c0c8340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00efc0 tx=0x7fd97c00c490 comp rx=0 tx=0).stop 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 shutdown_connections 2024-04-25T14:07:02.515 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fd9780744b0 0x7fd9780769c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fd98c0b91f0 0x7fd98c14a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fd98c0b4c10 0x7fd98c0c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 --2- 172.21.15.150:0/689020706 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fd98c06d800 0x7fd98c0c8340 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 >> 172.21.15.150:0/689020706 conn(0x7fd98c0b7690 msgr2=0x7fd98c15dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 shutdown_connections 2024-04-25T14:07:02.516 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.514+0000 7fd9918d1640 1 -- 172.21.15.150:0/689020706 wait complete. 2024-04-25T14:07:02.525 INFO:tasks.ceph:Scrubbing osd.3 2024-04-25T14:07:02.525 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:02.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.637+0000 7fc9f70bd640 1 Processor -- start 2024-04-25T14:07:02.637 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.637+0000 7fc9f70bd640 1 -- start start 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f70bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f006ccf0 0x7fc9f006d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f70bd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9f0160270 0x7fc9f0160680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f70bd640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9f006d640 con 0x7fc9f0160bc0 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f70bd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc9f0160fd0 con 0x7fc9f0160270 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f70bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f0161210 con 0x7fc9f006ccf0 2024-04-25T14:07:02.640 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9f0160270 0x7fc9f0160680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f565a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fc9f0160bc0 0x7fc9f0163d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46444/0 (socket says 172.21.15.150:46444) 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc9f0160270 0x7fc9f0160680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33644/0 (socket says 172.21.15.150:33644) 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f565a640 1 -- 172.21.15.150:0/3855309125 learned_addr learned my addr 172.21.15.150:0/3855309125 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9f4e59640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f006ccf0 0x7fc9f006d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 -- 172.21.15.150:0/3855309125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f006ccf0 msgr2=0x7fc9f006d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f006ccf0 0x7fc9f006d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 -- 172.21.15.150:0/3855309125 >> v1:172.21.15.150:6789/0 conn(0x7fc9f0160bc0 legacy=0x7fc9f0163d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.641 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9effff640 1 -- 172.21.15.150:0/3855309125 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f01614f0 con 0x7fc9f0160270 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.638+0000 7fc9ef7fe640 1 -- 172.21.15.150:0/3855309125 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4265618332 0 0) 0x7fc9e8001490 con 0x7fc9f0160bc0 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.640+0000 7fc9effff640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f0160270 0x7fc9f0160680 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097a0 tx=0x7fc9e402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753fe50216a9f917 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.640+0000 7fc9ef7fe640 1 -- 172.21.15.150:0/3855309125 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4037070 con 0x7fc9f0160270 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.640+0000 7fc9ef7fe640 1 -- 172.21.15.150:0/3855309125 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e4032440 con 0x7fc9f0160270 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.640+0000 7fc9ef7fe640 1 -- 172.21.15.150:0/3855309125 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4032810 con 0x7fc9f0160270 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 -- 172.21.15.150:0/3855309125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f0160270 msgr2=0x7fc9f0160680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f0160270 0x7fc9f0160680 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc9e40097a0 tx=0x7fc9e402a180 comp rx=0 tx=0).stop 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 -- 172.21.15.150:0/3855309125 shutdown_connections 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f0160270 0x7fc9f0160680 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/3855309125 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f006ccf0 0x7fc9f006d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 -- 172.21.15.150:0/3855309125 >> 172.21.15.150:0/3855309125 conn(0x7fc9f0073da0 msgr2=0x7fc9f009b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 -- 172.21.15.150:0/3855309125 shutdown_connections 2024-04-25T14:07:02.642 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.641+0000 7fc9f70bd640 1 -- 172.21.15.150:0/3855309125 wait complete. 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.643+0000 7fc9f70bd640 1 Processor -- start 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.643+0000 7fc9f70bd640 1 -- start start 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 0x7fc9f00b1430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9f0160270 0x7fc9f0153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f0160bc0 0x7fc9f0154740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f0161d90 con 0x7fc9f006ccf0 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc9f0161ab0 con 0x7fc9f0160270 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f70bd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc9f00b7730 con 0x7fc9f0160bc0 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f4e59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 0x7fc9f00b1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f4e59640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 0x7fc9f00b1430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33650/0 (socket says 172.21.15.150:33650) 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f4e59640 1 -- 172.21.15.150:0/1051864906 learned_addr learned my addr 172.21.15.150:0/1051864906 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:02.647 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9f565a640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f0160bc0 0x7fc9f0154740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.644+0000 7fc9effff640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9f0160270 0x7fc9f0153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 -- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f0160bc0 msgr2=0x7fc9f0154740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f0160bc0 0x7fc9f0154740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 -- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9f0160270 msgr2=0x7fc9f0153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc9f0160270 0x7fc9f0153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9f00b0670 con 0x7fc9f006ccf0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f565a640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc9f0160bc0 0x7fc9f0154740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f4e59640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 0x7fc9f00b1430 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d620 tx=0x7fc9e000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000b620 con 0x7fc9f006ccf0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc9e0019070 con 0x7fc9f006ccf0 2024-04-25T14:07:02.648 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9f00b0950 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9e000ba00 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.645+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9f00b0f10 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.646+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc9f00b0420 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc9e000bba0 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9dc0744b0 0x7fc9dc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fc9e0095500 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9effff640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9dc0744b0 0x7fc9dc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7fc9dc07aa40 0x7fc9dc07ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc9e0013e50 con 0x7fc9dc07aa40 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc9e009c050 con 0x7fc9f006ccf0 2024-04-25T14:07:02.649 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.647+0000 7fc9f565a640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7fc9dc07aa40 0x7fc9dc07ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.648+0000 7fc9effff640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9dc0744b0 0x7fc9dc0769c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4034000 tx=0x7fc9e402e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.648+0000 7fc9f565a640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7fc9dc07aa40 0x7fc9dc07ced0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.650 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.648+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== osd.3 v2:172.21.15.150:6800/690753092 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc9e0013e50 con 0x7fc9dc07aa40 2024-04-25T14:07:02.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.661+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 --> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc9f0076a40 con 0x7fc9dc07aa40 2024-04-25T14:07:02.660 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9ed7fa640 1 -- 172.21.15.150:0/1051864906 <== osd.3 v2:172.21.15.150:6800/690753092 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc9f0076a40 con 0x7fc9dc07aa40 2024-04-25T14:07:02.661 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:02.661 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:02.661 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7fc9dc07aa40 msgr2=0x7fc9dc07ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6800/690753092,v1:172.21.15.150:6801/690753092] conn(0x7fc9dc07aa40 0x7fc9dc07ced0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9dc0744b0 msgr2=0x7fc9dc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc9dc0744b0 0x7fc9dc0769c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4034000 tx=0x7fc9e402e000 comp rx=0 tx=0).stop 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 msgr2=0x7fc9f00b1430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 --2- 172.21.15.150:0/1051864906 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc9f006ccf0 0x7fc9f00b1430 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fc9e000d620 tx=0x7fc9e000dae0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f565a640 1 -- 172.21.15.150:0/1051864906 reap_dead start 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 shutdown_connections 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 >> 172.21.15.150:0/1051864906 conn(0x7fc9f0073da0 msgr2=0x7fc9f01621b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.662 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 shutdown_connections 2024-04-25T14:07:02.663 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.662+0000 7fc9f70bd640 1 -- 172.21.15.150:0/1051864906 wait complete. 2024-04-25T14:07:02.672 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-25T14:07:02.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.782+0000 7f99cc870640 1 Processor -- start 2024-04-25T14:07:02.781 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.782+0000 7f99cc870640 1 -- start start 2024-04-25T14:07:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.783+0000 7f99cc870640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f99c806f640 0x7f99c806cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.783+0000 7f99cc870640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 0x7f99c809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.783 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.783+0000 7f99cc870640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c815e9c0 con 0x7f99c815e4e0 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.783+0000 7f99cc870640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f99c815eca0 con 0x7f99c806f640 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.783+0000 7f99cc870640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f99c815ef80 con 0x7f99c806d490 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c659c640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f99c815e4e0 0x7f99c80b2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46446/0 (socket says 172.21.15.150:46446) 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c659c640 1 -- 172.21.15.150:0/3330487656 learned_addr learned my addr 172.21.15.150:0/3330487656 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c6d9d640 1 --2- 172.21.15.150:0/3330487656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 0x7f99c809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99bffff640 1 --2- 172.21.15.150:0/3330487656 >> v2:172.21.15.187:3300/0 conn(0x7f99c806f640 0x7f99c806cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c5d9b640 1 -- 172.21.15.150:0/3330487656 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164978954 0 0) 0x7f99c815e9c0 con 0x7f99c815e4e0 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c5d9b640 1 -- 172.21.15.150:0/3330487656 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f99b4003520 con 0x7f99c815e4e0 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c6d9d640 1 -- 172.21.15.150:0/3330487656 >> v2:172.21.15.187:3300/0 conn(0x7f99c806f640 msgr2=0x7f99c806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c6d9d640 1 --2- 172.21.15.150:0/3330487656 >> v2:172.21.15.187:3300/0 conn(0x7f99c806f640 0x7f99c806cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c6d9d640 1 -- 172.21.15.150:0/3330487656 >> v1:172.21.15.150:6789/0 conn(0x7f99c815e4e0 legacy=0x7f99c80b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.784 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.784+0000 7f99c6d9d640 1 -- 172.21.15.150:0/3330487656 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c815f260 con 0x7f99c806d490 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.785+0000 7f99c6d9d640 1 --2- 172.21.15.150:0/3330487656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 0x7f99c809b4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009d90 tx=0x7f99c002a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1726bb4185917 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.785+0000 7f99c5d9b640 1 -- 172.21.15.150:0/3330487656 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0007740 con 0x7f99c806d490 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.785+0000 7f99c5d9b640 1 -- 172.21.15.150:0/3330487656 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99c00078e0 con 0x7f99c806d490 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.785+0000 7f99c5d9b640 1 -- 172.21.15.150:0/3330487656 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99c0007c90 con 0x7f99c806d490 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 -- 172.21.15.150:0/3330487656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 msgr2=0x7f99c809b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 --2- 172.21.15.150:0/3330487656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 0x7f99c809b4a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f99c0009d90 tx=0x7f99c002a660 comp rx=0 tx=0).stop 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 -- 172.21.15.150:0/3330487656 shutdown_connections 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 --2- 172.21.15.150:0/3330487656 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c806d490 0x7f99c809b4a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 --2- 172.21.15.150:0/3330487656 >> v2:172.21.15.187:3300/0 conn(0x7f99c806f640 0x7f99c806cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 -- 172.21.15.150:0/3330487656 >> 172.21.15.150:0/3330487656 conn(0x7f99c80b7680 msgr2=0x7f99c80b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.786 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 -- 172.21.15.150:0/3330487656 shutdown_connections 2024-04-25T14:07:02.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.786+0000 7f99cc870640 1 -- 172.21.15.150:0/3330487656 wait complete. 2024-04-25T14:07:02.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.788+0000 7f99cc870640 1 Processor -- start 2024-04-25T14:07:02.787 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.788+0000 7f99cc870640 1 -- start start 2024-04-25T14:07:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c806f640 0x7f99c815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c815e4e0 0x7f99c8159400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f99c809bdb0 con 0x7f99c806d490 2024-04-25T14:07:02.789 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f99c809bc30 con 0x7f99c806f640 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99cc870640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f99c80d2c20 con 0x7f99c815e4e0 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33654/0 (socket says 172.21.15.150:33654) 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 -- 172.21.15.150:0/143457156 learned_addr learned my addr 172.21.15.150:0/143457156 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c6d9d640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c815e4e0 0x7f99c8159400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99bffff640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c806f640 0x7f99c815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 -- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c815e4e0 msgr2=0x7f99c8159400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c815e4e0 0x7f99c8159400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 -- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c806f640 msgr2=0x7f99c815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c806f640 0x7f99c815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.789+0000 7f99c659c640 1 -- 172.21.15.150:0/143457156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c815b7f0 con 0x7f99c806d490 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99c659c640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f99b8001eb0 tx=0x7f99b800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.790 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8017070 con 0x7f99c806d490 2024-04-25T14:07:02.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b8005d00 con 0x7f99c806d490 2024-04-25T14:07:02.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c815bd00 con 0x7f99c806d490 2024-04-25T14:07:02.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8007310 con 0x7f99c806d490 2024-04-25T14:07:02.791 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.790+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c80afaa0 con 0x7f99c806d490 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.791+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c8076a40 con 0x7f99c806d490 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.791+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f99b801a070 con 0x7f99c806d490 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.792+0000 7f99beffd640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 0x7f99a80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.792+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f99b80944d0 con 0x7f99c806d490 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.792+0000 7f99bffff640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 0x7f99a80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:02.792 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.792+0000 7f99bffff640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 0x7f99a80769c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f99c80b1540 tx=0x7f99b0008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:02.795 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.796+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99b805ed80 con 0x7f99c806d490 2024-04-25T14:07:02.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.970+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f99c80d2340 con 0x7f99a80744b0 2024-04-25T14:07:02.969 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.970+0000 7f99beffd640 1 -- 172.21.15.150:0/143457156 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f99c80d2340 con 0x7f99a80744b0 2024-04-25T14:07:02.969 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 3 to deep-scrub 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 msgr2=0x7f99a80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 0x7f99a80769c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f99c80b1540 tx=0x7f99b0008040 comp rx=0 tx=0).stop 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 msgr2=0x7f99c81f63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f99b8001eb0 tx=0x7f99b800c820 comp rx=0 tx=0).stop 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 shutdown_connections 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f99a80744b0 0x7f99a80769c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f99c815e4e0 0x7f99c8159400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.972 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f99c806f640 0x7f99c815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 --2- 172.21.15.150:0/143457156 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f99c806d490 0x7f99c81f63d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:02.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 >> 172.21.15.150:0/143457156 conn(0x7f99c80b7680 msgr2=0x7f99c8070440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:02.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 shutdown_connections 2024-04-25T14:07:02.973 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:02.972+0000 7f99cc870640 1 -- 172.21.15.150:0/143457156 wait complete. 2024-04-25T14:07:02.982 INFO:tasks.ceph:Scrubbing osd.4 2024-04-25T14:07:02.983 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.095+0000 7fa813268640 1 Processor -- start 2024-04-25T14:07:03.094 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.095+0000 7fa813268640 1 -- start start 2024-04-25T14:07:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa80c163810 0x7fa80c0d16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c0b5420 0x7fa80c0b5830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa80c072cd0 0x7fa80c0730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.096 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa80c06e8f0 con 0x7fa80c163810 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa80c0ce5d0 con 0x7fa80c072cd0 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa813268640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c0b5d70 con 0x7fa80c0b5420 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa80c163810 0x7fa80c0d16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811805640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa80c072cd0 0x7fa80c0730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa80c163810 0x7fa80c0d16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33664/0 (socket says 172.21.15.150:33664) 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 -- 172.21.15.150:0/156602695 learned_addr learned my addr 172.21.15.150:0/156602695 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa810803640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c0b5420 0x7fa80c0b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 -- 172.21.15.150:0/156602695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c0b5420 msgr2=0x7fa80c0b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c0b5420 0x7fa80c0b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 -- 172.21.15.150:0/156602695 >> v2:172.21.15.187:3300/0 conn(0x7fa80c072cd0 msgr2=0x7fa80c0730e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 --2- 172.21.15.150:0/156602695 >> v2:172.21.15.187:3300/0 conn(0x7fa80c072cd0 0x7fa80c0730e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.097 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.097+0000 7fa811004640 1 -- 172.21.15.150:0/156602695 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c1641b0 con 0x7fa80c163810 2024-04-25T14:07:03.098 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.098+0000 7fa811004640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c163810 0x7fa80c0d16e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa80c073da0 tx=0x7fa80802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b7f80df92044a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.098+0000 7fa803fff640 1 -- 172.21.15.150:0/156602695 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808036070 con 0x7fa80c163810 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.098+0000 7fa803fff640 1 -- 172.21.15.150:0/156602695 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa808031600 con 0x7fa80c163810 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.098+0000 7fa803fff640 1 -- 172.21.15.150:0/156602695 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8080319d0 con 0x7fa80c163810 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 -- 172.21.15.150:0/156602695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c163810 msgr2=0x7fa80c0d16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c163810 0x7fa80c0d16e0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fa80c073da0 tx=0x7fa80802a610 comp rx=0 tx=0).stop 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 -- 172.21.15.150:0/156602695 shutdown_connections 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 --2- 172.21.15.150:0/156602695 >> v2:172.21.15.187:3300/0 conn(0x7fa80c072cd0 0x7fa80c0730e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c0b5420 0x7fa80c0b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 --2- 172.21.15.150:0/156602695 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c163810 0x7fa80c0d16e0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 -- 172.21.15.150:0/156602695 >> 172.21.15.150:0/156602695 conn(0x7fa80c0d0420 msgr2=0x7fa80c0d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 -- 172.21.15.150:0/156602695 shutdown_connections 2024-04-25T14:07:03.099 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.099+0000 7fa813268640 1 -- 172.21.15.150:0/156602695 wait complete. 2024-04-25T14:07:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.101+0000 7fa813268640 1 Processor -- start 2024-04-25T14:07:03.100 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.101+0000 7fa813268640 1 -- start start 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 0x7fa80c14ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa80c0b5420 0x7fa80c14f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c163810 0x7fa80c0cb580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c164a50 con 0x7fa80c072cd0 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c164770 con 0x7fa80c0b5420 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa813268640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c0d1c20 con 0x7fa80c163810 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811004640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 0x7fa80c14ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.103 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811004640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 0x7fa80c14ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33668/0 (socket says 172.21.15.150:33668) 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811004640 1 -- 172.21.15.150:0/73140962 learned_addr learned my addr 172.21.15.150:0/73140962 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811805640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c163810 0x7fa80c0cb580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa810803640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa80c0b5420 0x7fa80c14f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811004640 1 -- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c163810 msgr2=0x7fa80c0cb580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.102+0000 7fa811004640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c163810 0x7fa80c0cb580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa811004640 1 -- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa80c0b5420 msgr2=0x7fa80c14f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa811004640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa80c0b5420 0x7fa80c14f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa811004640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c0cafe0 con 0x7fa80c072cd0 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa810803640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa80c0b5420 0x7fa80c14f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa811805640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa80c163810 0x7fa80c0cb580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa811004640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 0x7fa80c14ecd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa808034040 tx=0x7fa808004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808036050 con 0x7fa80c072cd0 2024-04-25T14:07:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa808042070 con 0x7fa80c072cd0 2024-04-25T14:07:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80c162790 con 0x7fa80c072cd0 2024-04-25T14:07:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa808009510 con 0x7fa80c072cd0 2024-04-25T14:07:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.103+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa80c161cb0 con 0x7fa80c072cd0 2024-04-25T14:07:03.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.104+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa80c076a40 con 0x7fa80c072cd0 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.105+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa808007c50 con 0x7fa80c072cd0 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.105+0000 7fa801ffb640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa7f00744e0 0x7fa7f00769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.105+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fa8080b68f0 con 0x7fa80c072cd0 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.105+0000 7fa810803640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa7f00744e0 0x7fa7f00769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa801ffb640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7fa7f007aa70 0x7fa7f007cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.106 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7f007d5d0 con 0x7fa7f007aa70 2024-04-25T14:07:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fa8080b6d80 con 0x7fa80c072cd0 2024-04-25T14:07:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa811805640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7fa7f007aa70 0x7fa7f007cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa810803640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa7f00744e0 0x7fa7f00769f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009af0 tx=0x7fa7fc008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.106+0000 7fa811805640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7fa7f007aa70 0x7fa7f007cf00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.107+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== osd.4 v2:172.21.15.187:6824/361202506 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa7f007d5d0 con 0x7fa7f007aa70 2024-04-25T14:07:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.120+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 --> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa80c06e6d0 con 0x7fa7f007aa70 2024-04-25T14:07:03.119 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.120+0000 7fa801ffb640 1 -- 172.21.15.150:0/73140962 <== osd.4 v2:172.21.15.187:6824/361202506 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa80c06e6d0 con 0x7fa7f007aa70 2024-04-25T14:07:03.120 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:03.120 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:03.120 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:03.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7fa7f007aa70 msgr2=0x7fa7f007cf00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.187:6824/361202506,v1:172.21.15.187:6825/361202506] conn(0x7fa7f007aa70 0x7fa7f007cf00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa7f00744e0 msgr2=0x7fa7f00769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.121 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa7f00744e0 0x7fa7f00769f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc009af0 tx=0x7fa7fc008040 comp rx=0 tx=0).stop 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 msgr2=0x7fa80c14ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 --2- 172.21.15.150:0/73140962 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa80c072cd0 0x7fa80c14ecd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa808034040 tx=0x7fa808004a40 comp rx=0 tx=0).stop 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa811805640 1 -- 172.21.15.150:0/73140962 reap_dead start 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 shutdown_connections 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 >> 172.21.15.150:0/73140962 conn(0x7fa80c0d0420 msgr2=0x7fa80c06f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 shutdown_connections 2024-04-25T14:07:03.122 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.121+0000 7fa813268640 1 -- 172.21.15.150:0/73140962 wait complete. 2024-04-25T14:07:03.131 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-25T14:07:03.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.240+0000 7f1b14fce640 1 Processor -- start 2024-04-25T14:07:03.239 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.240+0000 7f1b14fce640 1 -- start start 2024-04-25T14:07:03.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b14fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b10160150 0x7f1b100b91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b14fce640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b1006d800 0x7f1b100d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b14fce640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b100d4f20 con 0x7f1b100b4c10 2024-04-25T14:07:03.241 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b14fce640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f1b1006cc60 con 0x7f1b1006d800 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b14fce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b10160560 con 0x7f1b10160150 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b0ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b1006d800 0x7f1b100d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b0ed9d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f1b1006d800 0x7f1b100d0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33674/0 (socket says 172.21.15.150:33674) 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b0ed9d640 1 -- 172.21.15.150:0/2278708770 learned_addr learned my addr 172.21.15.150:0/2278708770 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.241+0000 7f1b0dd9b640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b10160150 0x7f1b100b91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0ed9d640 1 -- 172.21.15.150:0/2278708770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b10160150 msgr2=0x7f1b100b91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0ed9d640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b10160150 0x7f1b100b91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0ed9d640 1 -- 172.21.15.150:0/2278708770 >> v1:172.21.15.187:6789/0 conn(0x7f1b100b4c10 legacy=0x7f1b1006d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.242 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0ed9d640 1 -- 172.21.15.150:0/2278708770 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b100d5200 con 0x7f1b1006d800 2024-04-25T14:07:03.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0ed9d640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b1006d800 0x7f1b100d0950 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b0802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb1a8ff54d95d48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.243 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0d59a640 1 -- 172.21.15.150:0/2278708770 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b08037070 con 0x7f1b1006d800 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0d59a640 1 -- 172.21.15.150:0/2278708770 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b08032440 con 0x7f1b1006d800 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.242+0000 7f1b0d59a640 1 -- 172.21.15.150:0/2278708770 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b08032810 con 0x7f1b1006d800 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 -- 172.21.15.150:0/2278708770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b1006d800 msgr2=0x7f1b100d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b1006d800 0x7f1b100d0950 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009870 tx=0x7f1b0802a270 comp rx=0 tx=0).stop 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 -- 172.21.15.150:0/2278708770 shutdown_connections 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b1006d800 0x7f1b100d0950 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2278708770 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b10160150 0x7f1b100b91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 -- 172.21.15.150:0/2278708770 >> 172.21.15.150:0/2278708770 conn(0x7f1b100b7690 msgr2=0x7f1b100b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 -- 172.21.15.150:0/2278708770 shutdown_connections 2024-04-25T14:07:03.244 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.243+0000 7f1b14fce640 1 -- 172.21.15.150:0/2278708770 wait complete. 2024-04-25T14:07:03.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.245+0000 7f1b14fce640 1 Processor -- start 2024-04-25T14:07:03.245 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.245+0000 7f1b14fce640 1 -- start start 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b1006d800 0x7f1b100ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 0x7f1b101f9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1006c8f0 con 0x7f1b100b4c10 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f1b1006c610 con 0x7f1b10160150 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.246+0000 7f1b14fce640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1006dc10 con 0x7f1b1006d800 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33682/0 (socket says 172.21.15.150:33682) 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 -- 172.21.15.150:0/2218005563 learned_addr learned my addr 172.21.15.150:0/2218005563 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.247 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0e59c640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b1006d800 0x7f1b100ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0ed9d640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 0x7f1b101f9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 -- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b1006d800 msgr2=0x7f1b100ca4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b1006d800 0x7f1b100ca4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 -- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 msgr2=0x7f1b101f9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 0x7f1b101f9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 -- 172.21.15.150:0/2218005563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b100cc0d0 con 0x7f1b100b4c10 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0ed9d640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 0x7f1b101f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b0dd9b640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400e870 tx=0x7f1b0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b04017070 con 0x7f1b100b4c10 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b0400c940 con 0x7f1b100b4c10 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b100cbdf0 con 0x7f1b100b4c10 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b0400cd30 con 0x7f1b100b4c10 2024-04-25T14:07:03.248 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.247+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b101e9f30 con 0x7f1b100b4c10 2024-04-25T14:07:03.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.248+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b100cbb60 con 0x7f1b100b4c10 2024-04-25T14:07:03.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.249+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f1b0401a070 con 0x7f1b100b4c10 2024-04-25T14:07:03.249 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.249+0000 7f1afeffd640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 0x7f1af00769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.250+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f1b04095330 con 0x7f1b100b4c10 2024-04-25T14:07:03.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.250+0000 7f1b0e59c640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 0x7f1af00769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.250 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.250+0000 7f1b0e59c640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 0x7f1af00769c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1b00001ca0 tx=0x7f1b00009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.253 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.254+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b0405fc70 con 0x7f1b100b4c10 2024-04-25T14:07:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.429+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f1b1015de00 con 0x7f1af00744b0 2024-04-25T14:07:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.430+0000 7f1afeffd640 1 -- 172.21.15.150:0/2218005563 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f1b1015de00 con 0x7f1af00744b0 2024-04-25T14:07:03.429 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 4 to deep-scrub 2024-04-25T14:07:03.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 msgr2=0x7f1af00769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 0x7f1af00769c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1b00001ca0 tx=0x7f1b00009040 comp rx=0 tx=0).stop 2024-04-25T14:07:03.431 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 msgr2=0x7f1b101f9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f1b0400e870 tx=0x7f1b0400ed30 comp rx=0 tx=0).stop 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 shutdown_connections 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f1af00744b0 0x7f1af00769c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f1b10160150 0x7f1b101f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f1b100b4c10 0x7f1b101f9980 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 --2- 172.21.15.150:0/2218005563 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f1b1006d800 0x7f1b100ca4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 >> 172.21.15.150:0/2218005563 conn(0x7f1b100b7690 msgr2=0x7f1b100b5330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 shutdown_connections 2024-04-25T14:07:03.432 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.432+0000 7f1b14fce640 1 -- 172.21.15.150:0/2218005563 wait complete. 2024-04-25T14:07:03.442 INFO:tasks.ceph:Scrubbing osd.5 2024-04-25T14:07:03.442 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.555+0000 7f4f83dc5640 1 Processor -- start 2024-04-25T14:07:03.555 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.556+0000 7f4f83dc5640 1 -- start start 2024-04-25T14:07:03.557 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f83dc5640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c070d90 0x7f4f7c161510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f83dc5640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f7c0739c0 0x7f4f7c073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f83dc5640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c074310 con 0x7f4f7c15cd20 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f83dc5640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7c09f920 con 0x7f4f7c0739c0 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f83dc5640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c160a70 con 0x7f4f7c070d90 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f82362640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f4f7c0739c0 0x7f4f7c073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f81360640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f4f7c15cd20 0x7f4f7c0b06a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46456/0 (socket says 172.21.15.150:46456) 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f81360640 1 -- 172.21.15.150:0/1950254096 learned_addr learned my addr 172.21.15.150:0/1950254096 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f81b61640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c070d90 0x7f4f7c161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f82362640 1 -- 172.21.15.150:0/1950254096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c070d90 msgr2=0x7f4f7c161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:03.558 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f82362640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c070d90 0x7f4f7c161510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f82362640 1 -- 172.21.15.150:0/1950254096 >> v1:172.21.15.150:6789/0 conn(0x7f4f7c15cd20 legacy=0x7f4f7c0b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.558+0000 7f4f82362640 1 -- 172.21.15.150:0/1950254096 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c0b0ea0 con 0x7f4f7c0739c0 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.557+0000 7f4f80b5f640 1 -- 172.21.15.150:0/1950254096 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3330464729 0 0) 0x7f4f6c001110 con 0x7f4f7c15cd20 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.558+0000 7f4f82362640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c073dd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400ba70 tx=0x7f4f7402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee730e77cdfb95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.558+0000 7f4f80b5f640 1 -- 172.21.15.150:0/1950254096 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74032750 con 0x7f4f7c0739c0 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.558+0000 7f4f80b5f640 1 -- 172.21.15.150:0/1950254096 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f74030050 con 0x7f4f7c0739c0 2024-04-25T14:07:03.559 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.558+0000 7f4f80b5f640 1 -- 172.21.15.150:0/1950254096 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74032b10 con 0x7f4f7c0739c0 2024-04-25T14:07:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 -- 172.21.15.150:0/1950254096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0739c0 msgr2=0x7f4f7c073dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.560 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c073dd0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400ba70 tx=0x7f4f7402c890 comp rx=0 tx=0).stop 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 -- 172.21.15.150:0/1950254096 shutdown_connections 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c073dd0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/1950254096 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c070d90 0x7f4f7c161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 -- 172.21.15.150:0/1950254096 >> 172.21.15.150:0/1950254096 conn(0x7f4f7c0d1f80 msgr2=0x7f4f7c0732b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 -- 172.21.15.150:0/1950254096 shutdown_connections 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.559+0000 7f4f83dc5640 1 -- 172.21.15.150:0/1950254096 wait complete. 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.561+0000 7f4f83dc5640 1 Processor -- start 2024-04-25T14:07:03.561 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.561+0000 7f4f83dc5640 1 -- start start 2024-04-25T14:07:03.563 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 0x7f4f7c15fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c15cd20 0x7f4f7c1582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0b1180 con 0x7f4f7c070d90 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c160ff0 con 0x7f4f7c0739c0 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f83dc5640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c15d130 con 0x7f4f7c15cd20 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f81b61640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 0x7f4f7c15fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f81b61640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 0x7f4f7c15fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33704/0 (socket says 172.21.15.150:33704) 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.562+0000 7f4f81b61640 1 -- 172.21.15.150:0/4277333780 learned_addr learned my addr 172.21.15.150:0/4277333780 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81360640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 -- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c15cd20 msgr2=0x7f4f7c1582b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f82362640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c15cd20 0x7f4f7c1582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c15cd20 0x7f4f7c1582b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.564 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 -- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0739c0 msgr2=0x7f4f7c15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c15e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c15d5c0 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81360640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f4f7c0739c0 0x7f4f7c15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f82362640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f4f7c15cd20 0x7f4f7c1582b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f81b61640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 0x7f4f7c15fa10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009d10 tx=0x7f4f7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78018020 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f78014480 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78014830 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c15d8a0 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.563+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c158b50 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.564+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4f7c076a40 con 0x7f4f7c070d90 2024-04-25T14:07:03.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f4f78005d10 con 0x7f4f7c070d90 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4f600744b0 0x7f4f600769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f4f78096340 con 0x7f4f7c070d90 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f81360640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4f600744b0 0x7f4f600769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f4f6007aa40 0x7f4f6007ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f7801bec0 con 0x7f4f6007aa40 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.565+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f4f780967d0 con 0x7f4f7c070d90 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.566+0000 7f4f81360640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4f600744b0 0x7f4f600769c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.566+0000 7f4f82362640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f4f6007aa40 0x7f4f6007ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.566+0000 7f4f82362640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f4f6007aa40 0x7f4f6007ced0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.567+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== osd.5 v2:172.21.15.187:6800/3786766423 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4f7801bec0 con 0x7f4f6007aa40 2024-04-25T14:07:03.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.580+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 --> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4f7c160740 con 0x7f4f6007aa40 2024-04-25T14:07:03.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f727fc640 1 -- 172.21.15.150:0/4277333780 <== osd.5 v2:172.21.15.187:6800/3786766423 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f4f7c160740 con 0x7f4f6007aa40 2024-04-25T14:07:03.580 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:03.580 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:03.580 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f4f6007aa40 msgr2=0x7f4f6007ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.187:6800/3786766423,v1:172.21.15.187:6801/3786766423] conn(0x7f4f6007aa40 0x7f4f6007ced0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4f600744b0 msgr2=0x7f4f600769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f4f600744b0 0x7f4f600769c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000f60 tx=0x7f4f6c009040 comp rx=0 tx=0).stop 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 msgr2=0x7f4f7c15fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 --2- 172.21.15.150:0/4277333780 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f4f7c070d90 0x7f4f7c15fa10 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4f78009d10 tx=0x7f4f7800c490 comp rx=0 tx=0).stop 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f82362640 1 -- 172.21.15.150:0/4277333780 reap_dead start 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 shutdown_connections 2024-04-25T14:07:03.581 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 >> 172.21.15.150:0/4277333780 conn(0x7f4f7c0d1f80 msgr2=0x7f4f7c0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 shutdown_connections 2024-04-25T14:07:03.582 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.581+0000 7f4f83dc5640 1 -- 172.21.15.150:0/4277333780 wait complete. 2024-04-25T14:07:03.591 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-25T14:07:03.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.699+0000 7fb41c4b1640 1 Processor -- start 2024-04-25T14:07:03.698 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.699+0000 7fb41c4b1640 1 -- start start 2024-04-25T14:07:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41c4b1640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4140b4c10 0x7fb414160150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.700 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41c4b1640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fb41406d800 0x7fb4140d0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41c4b1640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4140b9600 con 0x7fb4140b91f0 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41c4b1640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fb41406cc60 con 0x7fb4140b4c10 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41c4b1640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fb4140d4f20 con 0x7fb41406d800 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41a24d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4140b4c10 0x7fb414160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41a24d640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fb4140b4c10 0x7fb414160150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33720/0 (socket says 172.21.15.150:33720) 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41a24d640 1 -- 172.21.15.150:0/3966589229 learned_addr learned my addr 172.21.15.150:0/3966589229 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.700+0000 7fb41aa4e640 1 --2- 172.21.15.150:0/3966589229 >> v2:172.21.15.187:3300/0 conn(0x7fb41406d800 0x7fb4140d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41a24d640 1 -- 172.21.15.150:0/3966589229 >> v2:172.21.15.187:3300/0 conn(0x7fb41406d800 msgr2=0x7fb4140d0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41a24d640 1 --2- 172.21.15.150:0/3966589229 >> v2:172.21.15.187:3300/0 conn(0x7fb41406d800 0x7fb4140d0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41a24d640 1 -- 172.21.15.150:0/3966589229 >> v1:172.21.15.187:6789/0 conn(0x7fb4140b91f0 legacy=0x7fb41406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.701 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41a24d640 1 -- 172.21.15.150:0/3966589229 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4140d5200 con 0x7fb4140b4c10 2024-04-25T14:07:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41a24d640 1 --2- 172.21.15.150:0/3966589229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb414160150 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb4140d0690 tx=0x7fb41002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea219a655fbc5423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41924b640 1 -- 172.21.15.150:0/3966589229 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410037070 con 0x7fb4140b4c10 2024-04-25T14:07:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41924b640 1 -- 172.21.15.150:0/3966589229 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb410032440 con 0x7fb4140b4c10 2024-04-25T14:07:03.702 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.701+0000 7fb41924b640 1 -- 172.21.15.150:0/3966589229 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb410032810 con 0x7fb4140b4c10 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 -- 172.21.15.150:0/3966589229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 msgr2=0x7fb414160150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/3966589229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb414160150 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb4140d0690 tx=0x7fb41002a270 comp rx=0 tx=0).stop 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 -- 172.21.15.150:0/3966589229 shutdown_connections 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/3966589229 >> v2:172.21.15.187:3300/0 conn(0x7fb41406d800 0x7fb4140d0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/3966589229 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb414160150 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.703 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 -- 172.21.15.150:0/3966589229 >> 172.21.15.150:0/3966589229 conn(0x7fb4140b7690 msgr2=0x7fb4140b7aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 -- 172.21.15.150:0/3966589229 shutdown_connections 2024-04-25T14:07:03.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.702+0000 7fb41c4b1640 1 -- 172.21.15.150:0/3966589229 wait complete. 2024-04-25T14:07:03.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.704+0000 7fb41c4b1640 1 Processor -- start 2024-04-25T14:07:03.704 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.704+0000 7fb41c4b1640 1 -- start start 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41406d800 0x7fb4141606f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4140b91f0 0x7fb41415d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fb41406c8f0 con 0x7fb4140b4c10 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fb41406c610 con 0x7fb4140b91f0 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb41c4b1640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fb41406dc10 con 0x7fb41406d800 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb419a4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.707 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb419a4c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33734/0 (socket says 172.21.15.150:33734) 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.705+0000 7fb419a4c640 1 -- 172.21.15.150:0/176598088 learned_addr learned my addr 172.21.15.150:0/176598088 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb41a24d640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41406d800 0x7fb4141606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb41aa4e640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4140b91f0 0x7fb41415d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 -- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41406d800 msgr2=0x7fb4141606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41406d800 0x7fb4141606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 -- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4140b91f0 msgr2=0x7fb41415d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4140b91f0 0x7fb41415d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 -- 172.21.15.150:0/176598088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb414161060 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb419a4c640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c440 tx=0x7fb40400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4040052d0 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb404005470 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb404007260 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb414161340 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.706+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb414161d90 con 0x7fb4140b4c10 2024-04-25T14:07:03.708 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.707+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb414076a40 con 0x7fb4140b4c10 2024-04-25T14:07:03.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.708+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fb4040074c0 con 0x7fb4140b4c10 2024-04-25T14:07:03.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.708+0000 7fb40affd640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:03.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.708+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fb404094440 con 0x7fb4140b4c10 2024-04-25T14:07:03.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.708+0000 7fb41a24d640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:03.709 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.709+0000 7fb41a24d640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 0x7fb3f80769c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb410034000 tx=0x7fb41002e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:03.711 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.713+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb40405e6a0 con 0x7fb4140b4c10 2024-04-25T14:07:03.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.885+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fb4140d4030 con 0x7fb3f80744b0 2024-04-25T14:07:03.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.886+0000 7fb40affd640 1 -- 172.21.15.150:0/176598088 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fb4140d4030 con 0x7fb3f80744b0 2024-04-25T14:07:03.885 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 5 to deep-scrub 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 msgr2=0x7fb3f80769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 0x7fb3f80769c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb410034000 tx=0x7fb41002e000 comp rx=0 tx=0).stop 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 msgr2=0x7fb4141f6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fb40400c440 tx=0x7fb40400c900 comp rx=0 tx=0).stop 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 shutdown_connections 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fb3f80744b0 0x7fb3f80769c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fb4140b91f0 0x7fb41415d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fb4140b4c10 0x7fb4141f6220 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 --2- 172.21.15.150:0/176598088 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fb41406d800 0x7fb4141606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:03.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 >> 172.21.15.150:0/176598088 conn(0x7fb4140b7690 msgr2=0x7fb4140b97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:03.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 shutdown_connections 2024-04-25T14:07:03.888 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:03.888+0000 7fb41c4b1640 1 -- 172.21.15.150:0/176598088 wait complete. 2024-04-25T14:07:03.898 INFO:tasks.ceph:Scrubbing osd.6 2024-04-25T14:07:03.898 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:04.009 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.010+0000 7f7eff8ee640 1 Processor -- start 2024-04-25T14:07:04.010 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.010+0000 7f7eff8ee640 1 -- start start 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eff8ee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 0x7f7ef80d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eff8ee640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f7ef80d1800 0x7f7ef8163eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eff8ee640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ef8160a10 con 0x7f7ef806ec00 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eff8ee640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f7ef81607b0 con 0x7f7ef80d1800 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eff8ee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef8160cb0 con 0x7f7ef80b5b30 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efd68a640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f7ef806ec00 0x7f7ef80b5420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46458/0 (socket says 172.21.15.150:46458) 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efd68a640 1 -- 172.21.15.150:0/4207568248 learned_addr learned my addr 172.21.15.150:0/4207568248 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efce89640 1 --2- 172.21.15.150:0/4207568248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 0x7f7ef80d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efde8b640 1 --2- 172.21.15.150:0/4207568248 >> v2:172.21.15.187:3300/0 conn(0x7f7ef80d1800 0x7f7ef8163eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.012 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eeffff640 1 -- 172.21.15.150:0/4207568248 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3743749155 0 0) 0x7f7ef8160a10 con 0x7f7ef806ec00 2024-04-25T14:07:04.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7eeffff640 1 -- 172.21.15.150:0/4207568248 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ee4003520 con 0x7f7ef806ec00 2024-04-25T14:07:04.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efce89640 1 -- 172.21.15.150:0/4207568248 >> v2:172.21.15.187:3300/0 conn(0x7f7ef80d1800 msgr2=0x7f7ef8163eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efce89640 1 --2- 172.21.15.150:0/4207568248 >> v2:172.21.15.187:3300/0 conn(0x7f7ef80d1800 0x7f7ef8163eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efce89640 1 -- 172.21.15.150:0/4207568248 >> v1:172.21.15.150:6789/0 conn(0x7f7ef806ec00 legacy=0x7f7ef80b5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.013 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.012+0000 7f7efce89640 1 -- 172.21.15.150:0/4207568248 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef8160f90 con 0x7f7ef80b5b30 2024-04-25T14:07:04.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.013+0000 7f7efce89640 1 --2- 172.21.15.150:0/4207568248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 0x7f7ef80d2710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7ee80097a0 tx=0x7f7ee802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=444a66bdd015f7ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.013+0000 7f7eeffff640 1 -- 172.21.15.150:0/4207568248 <== mon.2 v2:172.21.15.187:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee8037070 con 0x7f7ef80b5b30 2024-04-25T14:07:04.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.013+0000 7f7eeffff640 1 -- 172.21.15.150:0/4207568248 <== mon.2 v2:172.21.15.187:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ee8032440 con 0x7f7ef80b5b30 2024-04-25T14:07:04.014 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.013+0000 7f7eeffff640 1 -- 172.21.15.150:0/4207568248 <== mon.2 v2:172.21.15.187:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ee80327f0 con 0x7f7ef80b5b30 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 -- 172.21.15.150:0/4207568248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 msgr2=0x7f7ef80d2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/4207568248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 0x7f7ef80d2710 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7ee80097a0 tx=0x7f7ee802a180 comp rx=0 tx=0).stop 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 -- 172.21.15.150:0/4207568248 shutdown_connections 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/4207568248 >> v2:172.21.15.187:3300/0 conn(0x7f7ef80d1800 0x7f7ef8163eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/4207568248 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80b5b30 0x7f7ef80d2710 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 -- 172.21.15.150:0/4207568248 >> 172.21.15.150:0/4207568248 conn(0x7f7ef80d0420 msgr2=0x7f7ef80d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 -- 172.21.15.150:0/4207568248 shutdown_connections 2024-04-25T14:07:04.015 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.014+0000 7f7eff8ee640 1 -- 172.21.15.150:0/4207568248 wait complete. 2024-04-25T14:07:04.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.016+0000 7f7eff8ee640 1 Processor -- start 2024-04-25T14:07:04.016 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.016+0000 7f7eff8ee640 1 -- start start 2024-04-25T14:07:04.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 0x7f7ef80b6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ef80d1800 0x7f7ef81f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80cbf70 0x7f7ef80cc380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.017 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef806fd70 con 0x7f7ef806ec00 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f7ef8161550 con 0x7f7ef80d1800 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7eff8ee640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f7ef806fef0 con 0x7f7ef80cbf70 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7efd68a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 0x7f7ef80b6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7efd68a640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 0x7f7ef80b6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33746/0 (socket says 172.21.15.150:33746) 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7efd68a640 1 -- 172.21.15.150:0/1824676685 learned_addr learned my addr 172.21.15.150:0/1824676685 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7efce89640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ef80d1800 0x7f7ef81f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.017+0000 7f7efde8b640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80cbf70 0x7f7ef80cc380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 -- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80cbf70 msgr2=0x7f7ef80cc380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80cbf70 0x7f7ef80cc380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 -- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ef80d1800 msgr2=0x7f7ef81f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.018 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ef80d1800 0x7f7ef81f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ef80cc8c0 con 0x7f7ef806ec00 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efde8b640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f7ef80cbf70 0x7f7ef80cc380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efce89640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f7ef80d1800 0x7f7ef81f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7efd68a640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 0x7f7ef80b6a60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400ca50 tx=0x7f7ef400cf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4017070 con 0x7f7ef806ec00 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef400abb0 con 0x7f7ef806ec00 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ef8150f80 con 0x7f7ef806ec00 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4012400 con 0x7f7ef806ec00 2024-04-25T14:07:04.019 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.018+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ef8151540 con 0x7f7ef806ec00 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.019+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7ef814e690 con 0x7f7ef806ec00 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f7ef401a070 con 0x7f7ef806ec00 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7edc0744b0 0x7f7edc0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f7ef4094f20 con 0x7f7ef806ec00 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7efce89640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7edc0744b0 0x7f7edc0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7f7edc07aa40 0x7f7edc07ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ef4015e70 con 0x7f7edc07aa40 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.020+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7f7ef4097620 con 0x7f7ef806ec00 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.021+0000 7f7efce89640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7edc0744b0 0x7f7edc0769c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8034000 tx=0x7f7ee802e000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.021+0000 7f7efde8b640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7f7edc07aa40 0x7f7edc07ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.021 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.021+0000 7f7efde8b640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7f7edc07aa40 0x7f7edc07ced0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.022 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.022+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== osd.6 v2:172.21.15.187:6808/3350780520 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ef4015e70 con 0x7f7edc07aa40 2024-04-25T14:07:04.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.035+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 --> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f7ef8076a40 con 0x7f7edc07aa40 2024-04-25T14:07:04.034 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eedffb640 1 -- 172.21.15.150:0/1824676685 <== osd.6 v2:172.21.15.187:6808/3350780520 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f7ef8076a40 con 0x7f7edc07aa40 2024-04-25T14:07:04.035 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:04.035 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:04.035 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:04.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7f7edc07aa40 msgr2=0x7f7edc07ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.036 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.187:6808/3350780520,v1:172.21.15.187:6809/3350780520] conn(0x7f7edc07aa40 0x7f7edc07ced0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7edc0744b0 msgr2=0x7f7edc0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f7edc0744b0 0x7f7edc0769c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ee8034000 tx=0x7f7ee802e000 comp rx=0 tx=0).stop 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 msgr2=0x7f7ef80b6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 --2- 172.21.15.150:0/1824676685 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f7ef806ec00 0x7f7ef80b6a60 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400ca50 tx=0x7f7ef400cf10 comp rx=0 tx=0).stop 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7efde8b640 1 -- 172.21.15.150:0/1824676685 reap_dead start 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 shutdown_connections 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 >> 172.21.15.150:0/1824676685 conn(0x7f7ef80d0420 msgr2=0x7f7ef80b57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 shutdown_connections 2024-04-25T14:07:04.037 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.036+0000 7f7eff8ee640 1 -- 172.21.15.150:0/1824676685 wait complete. 2024-04-25T14:07:04.046 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-25T14:07:04.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.154+0000 7f299f3eb640 1 Processor -- start 2024-04-25T14:07:04.154 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.154+0000 7f299f3eb640 1 -- start start 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299f3eb640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f29981609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299f3eb640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29980cf300 con 0x7f29980d03a0 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299f3eb640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29980cf5e0 con 0x7f29980cdca0 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299f3eb640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f29980cff10 con 0x7f2998070820 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299d187640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f29980d03a0 0x7f29980cbd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:46466/0 (socket says 172.21.15.150:46466) 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299d187640 1 -- 172.21.15.150:0/1877471113 learned_addr learned my addr 172.21.15.150:0/1877471113 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:04.158 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232502027 0 0) 0x7f29980cf300 con 0x7f29980d03a0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f299c986640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f29981609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2984003520 con 0x7f29980d03a0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 313077419 0 0) 0x7f2984003520 con 0x7f29980d03a0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 msgr2=0x7f29981609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.156+0000 7f298ffff640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f29981609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 >> v1:172.21.15.187:6789/0 conn(0x7f29980cdca0 legacy=0x7f299806f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29980cf8c0 con 0x7f29980d03a0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7f29940019f0 con 0x7f29980d03a0 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 >> v1:172.21.15.150:6789/0 conn(0x7f29980d03a0 legacy=0x7f29980cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f2984003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2984004800 0x7f2984004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.159 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f2984005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f29980cff10 con 0x7f29840039e0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f298400dd00 con 0x7f2984004800 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f298400df00 con 0x7f2998070820 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f2984003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d988640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f2984005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299c986640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2984004800 0x7f2984004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 -- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 msgr2=0x7f2984005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998070820 0x7f2984005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 -- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2984004800 msgr2=0x7f2984004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2984004800 0x7f2984004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.157+0000 7f299d187640 1 -- 172.21.15.150:0/1877471113 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2984003520 con 0x7f29840039e0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.158+0000 7f299d187640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f2984003e50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f29980b0fa0 tx=0x7f2984005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f87a96548e6ac4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.158+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299400a480 con 0x7f29840039e0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.158+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f299402f070 con 0x7f29840039e0 2024-04-25T14:07:04.160 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.158+0000 7f298ffff640 1 -- 172.21.15.150:0/1877471113 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299402bc70 con 0x7f29840039e0 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 -- 172.21.15.150:0/1877471113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 msgr2=0x7f2984003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 --2- 172.21.15.150:0/1877471113 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f2984003e50 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f29980b0fa0 tx=0x7f2984005830 comp rx=0 tx=0).stop 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299d988640 1 -- 172.21.15.150:0/1877471113 reap_dead start 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 -- 172.21.15.150:0/1877471113 shutdown_connections 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 -- 172.21.15.150:0/1877471113 >> 172.21.15.150:0/1877471113 conn(0x7f29980ce390 msgr2=0x7f29980ce7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 -- 172.21.15.150:0/1877471113 shutdown_connections 2024-04-25T14:07:04.161 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.159+0000 7f299f3eb640 1 -- 172.21.15.150:0/1877471113 wait complete. 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.160+0000 7f299f3eb640 1 Processor -- start 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.160+0000 7f299f3eb640 1 -- start start 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2998153630 0x7f29980b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998153a40 0x7f29980b2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f299806f8a0 con 0x7f29840039e0 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f299806f720 con 0x7f2998153630 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299f3eb640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f299806fa20 con 0x7f2998153a40 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.164 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33768/0 (socket says 172.21.15.150:33768) 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 -- 172.21.15.150:0/2423706129 learned_addr learned my addr 172.21.15.150:0/2423706129 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d988640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998153a40 0x7f29980b2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299c986640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2998153630 0x7f29980b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 -- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998153a40 msgr2=0x7f29980b2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998153a40 0x7f29980b2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 -- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2998153630 msgr2=0x7f29980b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2998153630 0x7f29980b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 -- 172.21.15.150:0/2423706129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2998151a80 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f299d187640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f299402b4a0 tx=0x7f29980cfe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.162+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2994044070 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.163+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f299402bdb0 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.163+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2998151f80 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.163+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299400a480 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.163+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2998152540 con 0x7f29840039e0 2024-04-25T14:07:04.165 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.164+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998076a40 con 0x7f29840039e0 2024-04-25T14:07:04.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.164+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f2994047070 con 0x7f29840039e0 2024-04-25T14:07:04.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.165+0000 7f298dffb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 0x7f297c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.165+0000 7f299c986640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 0x7f297c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.165+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f29940b4d80 con 0x7f29840039e0 2024-04-25T14:07:04.166 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.165+0000 7f299c986640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 0x7f297c0769f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f29980b6680 tx=0x7f2988009190 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.168 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.169+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f299407efe0 con 0x7f29840039e0 2024-04-25T14:07:04.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.343+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f2998150800 con 0x7f297c0744e0 2024-04-25T14:07:04.342 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.343+0000 7f298dffb640 1 -- 172.21.15.150:0/2423706129 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f2988014b00 con 0x7f297c0744e0 2024-04-25T14:07:04.342 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 6 to deep-scrub 2024-04-25T14:07:04.344 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 msgr2=0x7f297c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 0x7f297c0769f0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f29980b6680 tx=0x7f2988009190 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 msgr2=0x7f29981502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f299402b4a0 tx=0x7f29980cfe80 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 shutdown_connections 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f297c0744e0 0x7f297c0769f0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f2998153a40 0x7f29980b2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f2998153630 0x7f29980b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 --2- 172.21.15.150:0/2423706129 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f29840039e0 0x7f29981502c0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 >> 172.21.15.150:0/2423706129 conn(0x7f29980ce390 msgr2=0x7f29980cbc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 shutdown_connections 2024-04-25T14:07:04.345 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.345+0000 7f299f3eb640 1 -- 172.21.15.150:0/2423706129 wait complete. 2024-04-25T14:07:04.355 INFO:tasks.ceph:Scrubbing osd.7 2024-04-25T14:07:04.356 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-25T14:07:04.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.469+0000 7fc9a0d08640 1 Processor -- start 2024-04-25T14:07:04.468 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.469+0000 7fc9a0d08640 1 -- start start 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c0b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc99c0d0da0 0x7fc99c0d4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fc99c0d4c70 0x7fc99c0d5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fc99c0d55c0 con 0x7fc99c0d0da0 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fc99c06d7e0 con 0x7fc99c0d4c70 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc9a0d08640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc99c0d0b80 con 0x7fc99c06dc70 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc999d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc99c0d0da0 0x7fc99c0d4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc999d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fc99c0d0da0 0x7fc99c0d4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33772/0 (socket says 172.21.15.150:33772) 2024-04-25T14:07:04.470 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc999d9b640 1 -- 172.21.15.150:0/2250541733 learned_addr learned my addr 172.21.15.150:0/2250541733 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc99a59c640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c0b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.470+0000 7fc99ad9d640 1 --2- 172.21.15.150:0/2250541733 >> v2:172.21.15.187:3300/0 conn(0x7fc99c0d4c70 0x7fc99c0d5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 -- 172.21.15.150:0/2250541733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 msgr2=0x7fc99c0b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c0b5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 -- 172.21.15.150:0/2250541733 >> v2:172.21.15.187:3300/0 conn(0x7fc99c0d4c70 msgr2=0x7fc99c0d5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2250541733 >> v2:172.21.15.187:3300/0 conn(0x7fc99c0d4c70 0x7fc99c0d5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.471 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 -- 172.21.15.150:0/2250541733 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c0d14a0 con 0x7fc99c0d0da0 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d0da0 0x7fc99c0d4730 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0b8af0 tx=0x7fc99002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88dcbebe85980b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.471+0000 7fc99959a640 1 -- 172.21.15.150:0/2250541733 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990037070 con 0x7fc99c0d0da0 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc99959a640 1 -- 172.21.15.150:0/2250541733 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc990032440 con 0x7fc99c0d0da0 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc99959a640 1 -- 172.21.15.150:0/2250541733 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990032810 con 0x7fc99c0d0da0 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2250541733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d0da0 msgr2=0x7fc99c0d4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d0da0 0x7fc99c0d4730 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0b8af0 tx=0x7fc99002a210 comp rx=0 tx=0).stop 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2250541733 shutdown_connections 2024-04-25T14:07:04.472 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2250541733 >> v2:172.21.15.187:3300/0 conn(0x7fc99c0d4c70 0x7fc99c0d5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d0da0 0x7fc99c0d4730 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2250541733 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c0b5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2250541733 >> 172.21.15.150:0/2250541733 conn(0x7fc99c0d2bc0 msgr2=0x7fc99c0d5b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2250541733 shutdown_connections 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.472+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2250541733 wait complete. 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.474+0000 7fc9a0d08640 1 Processor -- start 2024-04-25T14:07:04.473 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.474+0000 7fc9a0d08640 1 -- start start 2024-04-25T14:07:04.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c14c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c0d0da0 0x7fc99c150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.476 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 0x7fc99c14e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c0d11b0 con 0x7fc99c0d4c70 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fc99c06d470 con 0x7fc99c0d0da0 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fc99c163cb0 con 0x7fc99c06dc70 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 0x7fc99c14e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 0x7fc99c14e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33778/0 (socket says 172.21.15.150:33778) 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 -- 172.21.15.150:0/2024547964 learned_addr learned my addr 172.21.15.150:0/2024547964 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99a59c640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c14c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c0d0da0 0x7fc99c150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 -- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 msgr2=0x7fc99c14c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c14c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 -- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c0d0da0 msgr2=0x7fc99c150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fc99c0d0da0 0x7fc99c150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c1ecd90 con 0x7fc99c0d4c70 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99a59c640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fc99c06dc70 0x7fc99c14c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:04.477 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc99ad9d640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 0x7fc99c14e670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00e820 tx=0x7fc98c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c017070 con 0x7fc99c0d4c70 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c00ee70 con 0x7fc99c0d4c70 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c00cb60 con 0x7fc99c0d4c70 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.476+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc99c14cb70 con 0x7fc99c0d4c70 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.477+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc99c14d8b0 con 0x7fc99c0d4c70 2024-04-25T14:07:04.478 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.477+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc99c14e250 con 0x7fc99c0d4c70 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.478+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fc98c01a070 con 0x7fc99c0d4c70 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.478+0000 7fc98affd640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc97c0744b0 0x7fc97c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc99a59c640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc97c0744b0 0x7fc97c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fc98c095260 con 0x7fc99c0d4c70 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc98affd640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7fc97c07aa40 0x7fc97c07ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc98c016ec0 con 0x7fc97c07aa40 2024-04-25T14:07:04.479 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_get_version_reply(handle=1 version=65) ==== 24+0+0 (secure 0 0 0) 0x7fc98c0956f0 con 0x7fc99c0d4c70 2024-04-25T14:07:04.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc99a59c640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc97c0744b0 0x7fc97c0769c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc99c163c10 tx=0x7fc99400b040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7fc97c07aa40 0x7fc97c07ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.479+0000 7fc999d9b640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7fc97c07aa40 0x7fc97c07ced0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.480 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.480+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== osd.7 v2:172.21.15.187:6816/931266284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc98c016ec0 con 0x7fc97c07aa40 2024-04-25T14:07:04.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.494+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 --> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc99c14d380 con 0x7fc97c07aa40 2024-04-25T14:07:04.493 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.494+0000 7fc98affd640 1 -- 172.21.15.150:0/2024547964 <== osd.7 v2:172.21.15.187:6816/931266284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc99c14d380 con 0x7fc97c07aa40 2024-04-25T14:07:04.494 INFO:teuthology.orchestra.run.smithi150.stdout:{ 2024-04-25T14:07:04.494 INFO:teuthology.orchestra.run.smithi150.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-25T14:07:04.494 INFO:teuthology.orchestra.run.smithi150.stdout:} 2024-04-25T14:07:04.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7fc97c07aa40 msgr2=0x7fc97c07ced0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.187:6816/931266284,v1:172.21.15.187:6817/931266284] conn(0x7fc97c07aa40 0x7fc97c07ced0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc97c0744b0 msgr2=0x7fc97c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fc97c0744b0 0x7fc97c0769c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc99c163c10 tx=0x7fc99400b040 comp rx=0 tx=0).stop 2024-04-25T14:07:04.495 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 msgr2=0x7fc99c14e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 --2- 172.21.15.150:0/2024547964 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fc99c0d4c70 0x7fc99c14e670 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fc98c00e820 tx=0x7fc98c00ec20 comp rx=0 tx=0).stop 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc99ad9d640 1 -- 172.21.15.150:0/2024547964 reap_dead start 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 shutdown_connections 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 >> 172.21.15.150:0/2024547964 conn(0x7fc99c0d2bc0 msgr2=0x7fc99c0b9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 shutdown_connections 2024-04-25T14:07:04.496 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.495+0000 7fc9a0d08640 1 -- 172.21.15.150:0/2024547964 wait complete. 2024-04-25T14:07:04.505 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-25T14:07:04.614 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.615+0000 7faf309b2640 1 Processor -- start 2024-04-25T14:07:04.615 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.615+0000 7faf309b2640 1 -- start start 2024-04-25T14:07:04.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf309b2640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf2c070830 0x7faf2c0d2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf309b2640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7faf2c15e360 0x7faf2c0d25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf309b2640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf2c0d15d0 con 0x7faf2c0cf3d0 2024-04-25T14:07:04.616 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf309b2640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c0ce440 con 0x7faf2c070830 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf309b2640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7faf2c0d1870 con 0x7faf2c15e360 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf2c070830 0x7faf2c0d2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7faf2c070830 0x7faf2c0d2070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33794/0 (socket says 172.21.15.150:33794) 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 -- 172.21.15.150:0/3211936345 learned_addr learned my addr 172.21.15.150:0/3211936345 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf2ad9d640 1 --2- 172.21.15.150:0/3211936345 >> v2:172.21.15.187:3300/0 conn(0x7faf2c15e360 0x7faf2c0d25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 -- 172.21.15.150:0/3211936345 >> v2:172.21.15.187:3300/0 conn(0x7faf2c15e360 msgr2=0x7faf2c0d25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 --2- 172.21.15.150:0/3211936345 >> v2:172.21.15.187:3300/0 conn(0x7faf2c15e360 0x7faf2c0d25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 -- 172.21.15.150:0/3211936345 >> v1:172.21.15.150:6789/0 conn(0x7faf2c0cf3d0 legacy=0x7faf2c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.617 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.617+0000 7faf29d9b640 1 -- 172.21.15.150:0/3211936345 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c0d1b50 con 0x7faf2c070830 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.618+0000 7faf29d9b640 1 --2- 172.21.15.150:0/3211936345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c070830 0x7faf2c0d2070 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faf2c162e20 tx=0x7faf2002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e602e96afa4cf800 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.618+0000 7faf2959a640 1 -- 172.21.15.150:0/3211936345 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20037070 con 0x7faf2c070830 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.618+0000 7faf2959a640 1 -- 172.21.15.150:0/3211936345 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf20032440 con 0x7faf2c070830 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.618+0000 7faf2959a640 1 -- 172.21.15.150:0/3211936345 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20032810 con 0x7faf2c070830 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 -- 172.21.15.150:0/3211936345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c070830 msgr2=0x7faf2c0d2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 --2- 172.21.15.150:0/3211936345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c070830 0x7faf2c0d2070 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7faf2c162e20 tx=0x7faf2002a270 comp rx=0 tx=0).stop 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 -- 172.21.15.150:0/3211936345 shutdown_connections 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 --2- 172.21.15.150:0/3211936345 >> v2:172.21.15.187:3300/0 conn(0x7faf2c15e360 0x7faf2c0d25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 --2- 172.21.15.150:0/3211936345 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c070830 0x7faf2c0d2070 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 -- 172.21.15.150:0/3211936345 >> 172.21.15.150:0/3211936345 conn(0x7faf2c0d3960 msgr2=0x7faf2c0d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 -- 172.21.15.150:0/3211936345 shutdown_connections 2024-04-25T14:07:04.619 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.619+0000 7faf309b2640 1 -- 172.21.15.150:0/3211936345 wait complete. 2024-04-25T14:07:04.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.621+0000 7faf309b2640 1 Processor -- start 2024-04-25T14:07:04.620 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.621+0000 7faf309b2640 1 -- start start 2024-04-25T14:07:04.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf2c0cf3d0 0x7faf2c1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf2c146b30 0x7faf2c146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.622 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c070eb0 con 0x7faf2c15e360 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7faf2c070d30 con 0x7faf2c146b30 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf309b2640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7faf2c0cdfb0 con 0x7faf2c0cf3d0 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33810/0 (socket says 172.21.15.150:33810) 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 -- 172.21.15.150:0/548011831 learned_addr learned my addr 172.21.15.150:0/548011831 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf2a59c640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf2c0cf3d0 0x7faf2c1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf2ad9d640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf2c146b30 0x7faf2c146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 -- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf2c0cf3d0 msgr2=0x7faf2c1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf2c0cf3d0 0x7faf2c1582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 -- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf2c146b30 msgr2=0x7faf2c146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf2c146b30 0x7faf2c146380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.622+0000 7faf29d9b640 1 -- 172.21.15.150:0/548011831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2c1602f0 con 0x7faf2c15e360 2024-04-25T14:07:04.623 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf29d9b640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faf20000c00 tx=0x7faf200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20041020 con 0x7faf2c15e360 2024-04-25T14:07:04.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf200306c0 con 0x7faf2c15e360 2024-04-25T14:07:04.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2c143db0 con 0x7faf2c15e360 2024-04-25T14:07:04.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20037040 con 0x7faf2c15e360 2024-04-25T14:07:04.624 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.623+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf2c144370 con 0x7faf2c15e360 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.624+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf2c076a40 con 0x7faf2c15e360 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.624+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7faf20044020 con 0x7faf2c15e360 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.625+0000 7faf1affd640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 0x7faf0c0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.625+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7faf200b5900 con 0x7faf2c15e360 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.625+0000 7faf2a59c640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 0x7faf0c0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.625 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.625+0000 7faf2a59c640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 0x7faf0c0769f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faf1c001ca0 tx=0x7faf1c009040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.628 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.630+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faf2004b030 con 0x7faf2c15e360 2024-04-25T14:07:04.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.806+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7faf2c0b34a0 con 0x7faf0c0744e0 2024-04-25T14:07:04.806 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.807+0000 7faf1affd640 1 -- 172.21.15.150:0/548011831 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7faf2c0b34a0 con 0x7faf0c0744e0 2024-04-25T14:07:04.806 INFO:teuthology.orchestra.run.smithi150.stderr:instructed osd(s) 7 to deep-scrub 2024-04-25T14:07:04.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 msgr2=0x7faf0c0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 0x7faf0c0769f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7faf1c001ca0 tx=0x7faf1c009040 comp rx=0 tx=0).stop 2024-04-25T14:07:04.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 msgr2=0x7faf2c145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.808 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7faf20000c00 tx=0x7faf200046c0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 shutdown_connections 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7faf0c0744e0 0x7faf0c0769f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7faf2c146b30 0x7faf2c146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7faf2c15e360 0x7faf2c145e40 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 --2- 172.21.15.150:0/548011831 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7faf2c0cf3d0 0x7faf2c1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 >> 172.21.15.150:0/548011831 conn(0x7faf2c0d3960 msgr2=0x7faf2c0d3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 shutdown_connections 2024-04-25T14:07:04.809 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.809+0000 7faf309b2640 1 -- 172.21.15.150:0/548011831 wait complete. 2024-04-25T14:07:04.819 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:07:04.926 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.927+0000 7fa2fdccd640 1 Processor -- start 2024-04-25T14:07:04.927 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.927+0000 7fa2fdccd640 1 -- start start 2024-04-25T14:07:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fa2f80b7380 0x7fa2f815df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f80b3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f806ce40 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.929 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fa2f806c610 con 0x7fa2f80b7380 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2fdccd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f806c880 con 0x7fa2f815e450 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33826/0 (socket says 172.21.15.150:33826) 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 -- 172.21.15.150:0/4070489023 learned_addr learned my addr 172.21.15.150:0/4070489023 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f6ffd640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f80b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f77fe640 1 --2- 172.21.15.150:0/4070489023 >> v2:172.21.15.187:3300/0 conn(0x7fa2f80b7380 0x7fa2f815df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 -- 172.21.15.150:0/4070489023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 msgr2=0x7fa2f80b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f80b3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 -- 172.21.15.150:0/4070489023 >> v2:172.21.15.187:3300/0 conn(0x7fa2f80b7380 msgr2=0x7fa2f815df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 --2- 172.21.15.150:0/4070489023 >> v2:172.21.15.187:3300/0 conn(0x7fa2f80b7380 0x7fa2f815df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f7fff640 1 -- 172.21.15.150:0/4070489023 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f806cb60 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.929+0000 7fa2f6ffd640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f80b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.930+0000 7fa2f7fff640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa2f806ca90 tx=0x7fa2f002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f12f1e40d43f635 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.930 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.930+0000 7fa2f67fc640 1 -- 172.21.15.150:0/4070489023 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0036070 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.930+0000 7fa2f67fc640 1 -- 172.21.15.150:0/4070489023 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2f0031600 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.931 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.930+0000 7fa2f67fc640 1 -- 172.21.15.150:0/4070489023 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f00319d0 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 -- 172.21.15.150:0/4070489023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b3cd0 msgr2=0x7fa2f80b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fa2f806ca90 tx=0x7fa2f002a680 comp rx=0 tx=0).stop 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 -- 172.21.15.150:0/4070489023 shutdown_connections 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f80b40e0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/4070489023 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f80b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/4070489023 >> v2:172.21.15.187:3300/0 conn(0x7fa2f80b7380 0x7fa2f815df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 -- 172.21.15.150:0/4070489023 >> 172.21.15.150:0/4070489023 conn(0x7fa2f8073ce0 msgr2=0x7fa2f809b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 -- 172.21.15.150:0/4070489023 shutdown_connections 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.931+0000 7fa2fdccd640 1 -- 172.21.15.150:0/4070489023 wait complete. 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.933+0000 7fa2fdccd640 1 Processor -- start 2024-04-25T14:07:04.932 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.933+0000 7fa2fdccd640 1 -- start start 2024-04-25T14:07:04.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.934 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f8149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f815e860 con 0x7fa2f80b7380 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fa2f80b7880 con 0x7fa2f80b3cd0 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2fdccd640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fa2f815e9e0 con 0x7fa2f815e450 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2f6ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:33838/0 (socket says 172.21.15.150:33838) 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2f6ffd640 1 -- 172.21.15.150:0/659412541 learned_addr learned my addr 172.21.15.150:0/659412541 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2f7fff640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f8149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.934+0000 7fa2f77fe640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 -- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 msgr2=0x7fa2f8149860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f8149860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 -- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80b3cd0 msgr2=0x7fa2f815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:04.935 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f815ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 -- 172.21.15.150:0/659412541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2f80d12e0 con 0x7fa2f80b7380 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2f6ffd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00d800 tx=0x7fa2ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec004030 con 0x7fa2f80b7380 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2ec0041d0 con 0x7fa2f80b7380 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2f80d15c0 con 0x7fa2f80b7380 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec0096a0 con 0x7fa2f80b7380 2024-04-25T14:07:04.936 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.935+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2f81471e0 con 0x7fa2f80b7380 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.936+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2f815a690 con 0x7fa2f80b7380 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.937+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fa2ec009840 con 0x7fa2f80b7380 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.937+0000 7fa2dffff640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 0x7fa2e40769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.937+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fa2ec0951c0 con 0x7fa2f80b7380 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.937+0000 7fa2f77fe640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 0x7fa2e40769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:04.937 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.938+0000 7fa2f77fe640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 0x7fa2e40769c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8003610 tx=0x7fa2e8008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:04.941 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:04.942+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa2ec05f4b0 con 0x7fa2f80b7380 2024-04-25T14:07:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.114+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2f80d1790 con 0x7fa2e40744b0 2024-04-25T14:07:05.113 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.115+0000 7fa2dffff640 1 -- 172.21.15.150:0/659412541 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13459 (secure 0 0 0) 0x7fa2f80d1790 con 0x7fa2e40744b0 2024-04-25T14:07:05.114 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:07:05.114 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:07:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 msgr2=0x7fa2e40769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 0x7fa2e40769c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8003610 tx=0x7fa2e8008040 comp rx=0 tx=0).stop 2024-04-25T14:07:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 msgr2=0x7fa2f8146ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00d800 tx=0x7fa2ec00dcc0 comp rx=0 tx=0).stop 2024-04-25T14:07:05.116 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 shutdown_connections 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fa2e40744b0 0x7fa2e40769c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fa2f815e450 0x7fa2f8149860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fa2f80b7380 0x7fa2f8146ca0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 --2- 172.21.15.150:0/659412541 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fa2f80b3cd0 0x7fa2f815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 >> 172.21.15.150:0/659412541 conn(0x7fa2f8073ce0 msgr2=0x7fa2f815fb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 shutdown_connections 2024-04-25T14:07:05.117 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:05.117+0000 7fa2fdccd640 1 -- 172.21.15.150:0/659412541 wait complete. 2024-04-25T14:07:05.127 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-04-25T14:07:04.660203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":223488,"kb_used_data":7304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751232,"statfs":{"total":773094113280,"available":772865261568,"internally_reserved":0,"allocated":7479296,"data_stored":4070176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.637576"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":202,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T14:06:57.030186+0000","last_change":"2024-04-25T14:01:57.719890+0000","last_active":"2024-04-25T14:06:57.030186+0000","last_peered":"2024-04-25T14:06:57.030186+0000","last_clean":"2024-04-25T14:06:57.030186+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:06:57.030186+0000","last_undegraded":"2024-04-25T14:06:57.030186+0000","last_fullsized":"2024-04-25T14:06:57.030186+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T14:01:56.710022+0000","last_clean_scrub_stamp":"2024-04-25T14:01:56.710022+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:58:40.129916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1380352,"data_stored":951548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574912,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574914,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28368,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343472,"statfs":{"total":96636764160,"available":96607715328,"internally_reserved":0,"allocated":1380352,"data_stored":951548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:07:05.133 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-25T14:01:56.710022+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=14, tm_min=1, tm_sec=56, tm_wday=3, tm_yday=116, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=25, tm_hour=14, tm_min=7, tm_sec=0, tm_wday=3, tm_yday=116, tm_isdst=0) 2024-04-25T14:07:05.133 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-25T14:07:09.620 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:09.624 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:25.134 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T14:07:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.257+0000 7fcd2d434640 1 Processor -- start 2024-04-25T14:07:25.256 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.257+0000 7fcd2d434640 1 -- start start 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd2d434640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd28156cd0 0x7fcd2815d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd2d434640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fcd28070e70 0x7fcd2806ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd2d434640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd2806d750 con 0x7fcd2806d340 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd2d434640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7fcd28070250 con 0x7fcd28156cd0 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd2d434640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fcd280afe00 con 0x7fcd28070e70 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd26ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd28156cd0 0x7fcd2815d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd26ffd640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7fcd28156cd0 0x7fcd2815d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53424/0 (socket says 172.21.15.150:53424) 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.258+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1482550434 learned_addr learned my addr 172.21.15.150:0/1482550434 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:25.258 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd267fc640 1 --2- 172.21.15.150:0/1482550434 >> v2:172.21.15.187:3300/0 conn(0x7fcd28070e70 0x7fcd2806ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1482550434 >> v2:172.21.15.187:3300/0 conn(0x7fcd28070e70 msgr2=0x7fcd2806ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:25.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd26ffd640 1 --2- 172.21.15.150:0/1482550434 >> v2:172.21.15.187:3300/0 conn(0x7fcd28070e70 0x7fcd2806ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1482550434 >> v1:172.21.15.150:6789/0 conn(0x7fcd2806d340 legacy=0x7fcd28160180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd25ffb640 1 -- 172.21.15.150:0/1482550434 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2920525884 0 0) 0x7fcd20001490 con 0x7fcd2806d340 2024-04-25T14:07:25.259 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1482550434 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd280b00a0 con 0x7fcd28156cd0 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.259+0000 7fcd26ffd640 1 --2- 172.21.15.150:0/1482550434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28156cd0 0x7fcd2815d7a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fcd2815dce0 tx=0x7fcd1802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcacbfa3700eaa5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd25ffb640 1 -- 172.21.15.150:0/1482550434 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18037070 con 0x7fcd28156cd0 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd25ffb640 1 -- 172.21.15.150:0/1482550434 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd18032440 con 0x7fcd28156cd0 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd25ffb640 1 -- 172.21.15.150:0/1482550434 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18032810 con 0x7fcd28156cd0 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 -- 172.21.15.150:0/1482550434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28156cd0 msgr2=0x7fcd2815d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.260 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1482550434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28156cd0 0x7fcd2815d7a0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fcd2815dce0 tx=0x7fcd1802a1e0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 -- 172.21.15.150:0/1482550434 shutdown_connections 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1482550434 >> v2:172.21.15.187:3300/0 conn(0x7fcd28070e70 0x7fcd2806ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1482550434 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd28156cd0 0x7fcd2815d7a0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 -- 172.21.15.150:0/1482550434 >> 172.21.15.150:0/1482550434 conn(0x7fcd280d1ae0 msgr2=0x7fcd280d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 -- 172.21.15.150:0/1482550434 shutdown_connections 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.260+0000 7fcd2d434640 1 -- 172.21.15.150:0/1482550434 wait complete. 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.262+0000 7fcd2d434640 1 Processor -- start 2024-04-25T14:07:25.261 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.262+0000 7fcd2d434640 1 -- start start 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd28070e70 0x7fcd2815e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 0x7fcd2815e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fcd280b0940 con 0x7fcd2806d340 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fcd280b0660 con 0x7fcd28070e70 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd2d434640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fcd280b0e60 con 0x7fcd28156cd0 2024-04-25T14:07:25.264 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53436/0 (socket says 172.21.15.150:53436) 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1759485782 learned_addr learned my addr 172.21.15.150:0/1759485782 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd277fe640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 0x7fcd2815e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd267fc640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd28070e70 0x7fcd2815e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 msgr2=0x7fcd2815e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 0x7fcd2815e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd28070e70 msgr2=0x7fcd2815e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd28070e70 0x7fcd2815e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 -- 172.21.15.150:0/1759485782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd2809d9d0 con 0x7fcd2806d340 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd277fe640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 0x7fcd2815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd26ffd640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fcd1802a220 tx=0x7fcd180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.264+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18041020 con 0x7fcd2806d340 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.265+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd180306c0 con 0x7fcd2806d340 2024-04-25T14:07:25.265 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.265+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd18037040 con 0x7fcd2806d340 2024-04-25T14:07:25.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.265+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd2809dcb0 con 0x7fcd2806d340 2024-04-25T14:07:25.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.265+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd2809e270 con 0x7fcd2806d340 2024-04-25T14:07:25.266 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.266+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd28076a40 con 0x7fcd2806d340 2024-04-25T14:07:25.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.266+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fcd18044020 con 0x7fcd2806d340 2024-04-25T14:07:25.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.267+0000 7fcd0f7fe640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 0x7fcd140769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.267+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fcd180b59a0 con 0x7fcd2806d340 2024-04-25T14:07:25.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.267+0000 7fcd267fc640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 0x7fcd140769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.267 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.267+0000 7fcd267fc640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 0x7fcd140769f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006fd0 tx=0x7fcd1c008040 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.270 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.271+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd1804b030 con 0x7fcd2806d340 2024-04-25T14:07:25.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.443+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 --> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcd28154ac0 con 0x7fcd140744e0 2024-04-25T14:07:25.442 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.443+0000 7fcd0f7fe640 1 -- 172.21.15.150:0/1759485782 <== mgr.4110 v2:172.21.15.150:6832/3849822706 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13344 (secure 0 0 0) 0x7fcd28154ac0 con 0x7fcd140744e0 2024-04-25T14:07:25.443 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:07:25.443 INFO:teuthology.orchestra.run.smithi150.stderr:dumped all 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 msgr2=0x7fcd140769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 0x7fcd140769f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006fd0 tx=0x7fcd1c008040 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 msgr2=0x7fcd2813d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fcd1802a220 tx=0x7fcd180046c0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 shutdown_connections 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fcd140744e0 0x7fcd140769f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fcd28156cd0 0x7fcd2815e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fcd28070e70 0x7fcd2815e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 --2- 172.21.15.150:0/1759485782 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fcd2806d340 0x7fcd2813d2c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 >> 172.21.15.150:0/1759485782 conn(0x7fcd280d1ae0 msgr2=0x7fcd2806c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.445+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 shutdown_connections 2024-04-25T14:07:25.445 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.446+0000 7fcd2d434640 1 -- 172.21.15.150:0/1759485782 wait complete. 2024-04-25T14:07:25.456 INFO:teuthology.orchestra.run.smithi150.stdout:{"pg_ready":true,"pg_map":{"version":222,"stamp":"2024-04-25T14:07:24.664895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":223480,"kb_used_data":7304,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751240,"statfs":{"total":773094113280,"available":772865269760,"internally_reserved":0,"allocated":7479296,"data_stored":4070176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002963"},"pg_stats":[{"pgid":"1.0","version":"65'74","reported_seq":213,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-04-25T14:07:05.689913+0000","last_change":"2024-04-25T14:07:05.689955+0000","last_active":"2024-04-25T14:07:05.689913+0000","last_peered":"2024-04-25T14:07:05.689913+0000","last_clean":"2024-04-25T14:07:05.689913+0000","last_became_active":"2024-04-25T14:01:57.719531+0000","last_became_peered":"2024-04-25T14:01:57.719531+0000","last_unstale":"2024-04-25T14:07:05.689913+0000","last_undegraded":"2024-04-25T14:07:05.689913+0000","last_fullsized":"2024-04-25T14:07:05.689913+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'74","last_scrub_stamp":"2024-04-25T14:07:05.689822+0000","last_deep_scrub":"65'74","last_deep_scrub_stamp":"2024-04-25T14:07:05.689822+0000","last_clean_scrub_stamp":"2024-04-25T14:07:05.689822+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:14:03.821340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1380352,"data_stored":951548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574919,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574918,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":361180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1380352,"data_stored":951548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T14:07:25.457 DEBUG:teuthology.orchestra.run.smithi150:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-25T14:07:25.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.565+0000 7f86e048c640 1 Processor -- start 2024-04-25T14:07:25.565 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.565+0000 7f86e048c640 1 -- start start 2024-04-25T14:07:25.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86e048c640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7f86d806f640 0x7f86d806cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86e048c640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86d806d490 0x7f86d809b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86e048c640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86d815e9c0 con 0x7f86d815e4e0 2024-04-25T14:07:25.567 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86e048c640 1 -- --> v2:172.21.15.150:3300/0 -- mon_getmap magic: 0 -- 0x7f86d815eca0 con 0x7f86d806d490 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86e048c640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7f86d815ef80 con 0x7f86d806f640 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86dea29640 1 --2- >> v2:172.21.15.150:3300/0 conn(0x7f86d806d490 0x7f86d809b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86de228640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7f86d815e4e0 0x7f86d80b2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47376/0 (socket says 172.21.15.150:47376) 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86de228640 1 -- 172.21.15.150:0/1674010014 learned_addr learned my addr 172.21.15.150:0/1674010014 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.567+0000 7f86dda27640 1 --2- 172.21.15.150:0/1674010014 >> v2:172.21.15.187:3300/0 conn(0x7f86d806f640 0x7f86d806cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dea29640 1 -- 172.21.15.150:0/1674010014 >> v2:172.21.15.187:3300/0 conn(0x7f86d806f640 msgr2=0x7f86d806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dea29640 1 --2- 172.21.15.150:0/1674010014 >> v2:172.21.15.187:3300/0 conn(0x7f86d806f640 0x7f86d806cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dea29640 1 -- 172.21.15.150:0/1674010014 >> v1:172.21.15.150:6789/0 conn(0x7f86d815e4e0 legacy=0x7f86d80b2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dd226640 1 -- 172.21.15.150:0/1674010014 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 214176433 0 0) 0x7f86d815e9c0 con 0x7f86d815e4e0 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dea29640 1 -- 172.21.15.150:0/1674010014 --> v2:172.21.15.150:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d815f260 con 0x7f86d806d490 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dea29640 1 --2- 172.21.15.150:0/1674010014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806d490 0x7f86d809b4a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f86d8075470 tx=0x7f86d002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad533649aed80cb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dd226640 1 -- 172.21.15.150:0/1674010014 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0007740 con 0x7f86d806d490 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dd226640 1 -- 172.21.15.150:0/1674010014 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d00078e0 con 0x7f86d806d490 2024-04-25T14:07:25.568 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.568+0000 7f86dd226640 1 -- 172.21.15.150:0/1674010014 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0007cb0 con 0x7f86d806d490 2024-04-25T14:07:25.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 -- 172.21.15.150:0/1674010014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806d490 msgr2=0x7f86d809b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 --2- 172.21.15.150:0/1674010014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806d490 0x7f86d809b4a0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f86d8075470 tx=0x7f86d002a680 comp rx=0 tx=0).stop 2024-04-25T14:07:25.569 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 -- 172.21.15.150:0/1674010014 shutdown_connections 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 --2- 172.21.15.150:0/1674010014 >> v2:172.21.15.187:3300/0 conn(0x7f86d806f640 0x7f86d806cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 --2- 172.21.15.150:0/1674010014 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806d490 0x7f86d809b4a0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 -- 172.21.15.150:0/1674010014 >> 172.21.15.150:0/1674010014 conn(0x7f86d80b7680 msgr2=0x7f86d80b7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 -- 172.21.15.150:0/1674010014 shutdown_connections 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.569+0000 7f86e048c640 1 -- 172.21.15.150:0/1674010014 wait complete. 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.571+0000 7f86e048c640 1 Processor -- start 2024-04-25T14:07:25.570 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.571+0000 7f86e048c640 1 -- start start 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86d806d490 0x7f86d8157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d815e4e0 0x7f86d81e9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7f86d809bd70 con 0x7f86d806f640 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7f86d809bbf0 con 0x7f86d815e4e0 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86e048c640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7f86d80d2c20 con 0x7f86d806d490 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86dda27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86dda27640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53446/0 (socket says 172.21.15.150:53446) 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.572+0000 7f86dda27640 1 -- 172.21.15.150:0/3167477056 learned_addr learned my addr 172.21.15.150:0/3167477056 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86de228640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86d806d490 0x7f86d8157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dea29640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d815e4e0 0x7f86d81e9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.573 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 -- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86d806d490 msgr2=0x7f86d8157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86d806d490 0x7f86d8157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 -- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d815e4e0 msgr2=0x7f86d81e9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d815e4e0 0x7f86d81e9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 -- 172.21.15.150:0/3167477056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d8157db0 con 0x7f86d806f640 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86dda27640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f86c800d620 tx=0x7f86c800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c800b620 con 0x7f86d806f640 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c8019070 con 0x7f86d806f640 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c800ba00 con 0x7f86d806f640 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d8158d40 con 0x7f86d806f640 2024-04-25T14:07:25.574 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.573+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d8159300 con 0x7f86d806f640 2024-04-25T14:07:25.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.574+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d8076a40 con 0x7f86d806f640 2024-04-25T14:07:25.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.575+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7f86c800bba0 con 0x7f86d806f640 2024-04-25T14:07:25.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.575+0000 7f86ceffd640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 0x7f86bc0769f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.575+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7f86c8095410 con 0x7f86d806f640 2024-04-25T14:07:25.575 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.575+0000 7f86de228640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 0x7f86bc0769f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.576 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.576+0000 7f86de228640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 0x7f86bc0769f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86d4007fd0 tx=0x7f86d4007de0 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.579 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.580+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f86c805f660 con 0x7f86d806f640 2024-04-25T14:07:25.753 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.754+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f86d8158560 con 0x7f86d806f640 2024-04-25T14:07:25.756 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.757+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7f86c8093e20 con 0x7f86d806f640 2024-04-25T14:07:25.757 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.758+0000 7f86ceffd640 1 -- 172.21.15.150:0/3167477056 <== mon.0 v2:172.21.15.150:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f86c805f170 con 0x7f86d806f640 2024-04-25T14:07:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 msgr2=0x7f86bc0769f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 0x7f86bc0769f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f86d4007fd0 tx=0x7f86d4007de0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 msgr2=0x7f86d81e8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.759 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f86c800d620 tx=0x7f86c800dae0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 shutdown_connections 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7f86bc0744e0 0x7f86bc0769f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7f86d815e4e0 0x7f86d81e9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f86d806f640 0x7f86d81e8bd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 --2- 172.21.15.150:0/3167477056 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7f86d806d490 0x7f86d8157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 >> 172.21.15.150:0/3167477056 conn(0x7f86d80b7680 msgr2=0x7f86d80684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 shutdown_connections 2024-04-25T14:07:25.760 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.760+0000 7f86e048c640 1 -- 172.21.15.150:0/3167477056 wait complete. 2024-04-25T14:07:25.770 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T14:07:25.771 DEBUG:teuthology.orchestra.run.smithi150:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T14:07:25.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.880+0000 7fdfc0453640 1 Processor -- start 2024-04-25T14:07:25.879 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.880+0000 7fdfc0453640 1 -- start start 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfc0453640 1 --2- >> v2:172.21.15.187:3300/0 conn(0x7fdfb80d1120 0x7fdfb8070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfc0453640 1 -- --> v1:172.21.15.150:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb80aec30 con 0x7fdfb8161b10 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfc0453640 1 -- --> v1:172.21.15.187:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb80aef10 con 0x7fdfb8154950 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfc0453640 1 -- --> v2:172.21.15.187:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb80d1530 con 0x7fdfb80d1120 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd9ee640 1 --1- >> v1:172.21.15.150:6789/0 conn(0x7fdfb8161b10 0x7fdfb8161f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.150:6789/0 says I am v1:172.21.15.150:47388/0 (socket says 172.21.15.150:47388) 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd9ee640 1 -- 172.21.15.150:0/2884051295 learned_addr learned my addr 172.21.15.150:0/2884051295 (peer_addr_for_me v1:172.21.15.150:0/0) 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/2884051295 >> v2:172.21.15.187:3300/0 conn(0x7fdfb80d1120 0x7fdfb8070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v1:172.21.15.150:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 787180961 0 0) 0x7fdfb80aec30 con 0x7fdfb8161b10 2024-04-25T14:07:25.884 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 --> v1:172.21.15.150:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfa4003520 con 0x7fdfb8161b10 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v1:172.21.15.150:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3470129800 0 0) 0x7fdfa4003520 con 0x7fdfb8161b10 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 >> v2:172.21.15.187:3300/0 conn(0x7fdfb80d1120 msgr2=0x7fdfb8070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 --2- 172.21.15.150:0/2884051295 >> v2:172.21.15.187:3300/0 conn(0x7fdfb80d1120 0x7fdfb8070ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 >> v1:172.21.15.187:6789/0 conn(0x7fdfb8154950 legacy=0x7fdfb8154d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 --> v1:172.21.15.150:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb80af1f0 con 0x7fdfb8161b10 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.882+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v1:172.21.15.150:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1369041905 0 0) 0x7fdfa8001670 con 0x7fdfb8161b10 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 >> v1:172.21.15.150:6789/0 conn(0x7fdfb8161b10 legacy=0x7fdfb8161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 0x7fdfa4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa4004800 0x7fdfa4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa4005350 0x7fdfa400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80d1530 con 0x7fdfa40039b0 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfa400c280 con 0x7fdfa4005350 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfa400c480 con 0x7fdfa4004800 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 0x7fdfa4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.885 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbd9ee640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa4004800 0x7fdfa4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe9f0640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa4005350 0x7fdfa400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 -- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa4004800 msgr2=0x7fdfa4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa4004800 0x7fdfa4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 -- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa4005350 msgr2=0x7fdfa400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa4005350 0x7fdfa400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe1ef640 1 -- 172.21.15.150:0/2884051295 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfa4003520 con 0x7fdfa40039b0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe9f0640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfa4005350 0x7fdfa400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.883+0000 7fdfbe9f0640 1 -- 172.21.15.150:0/2884051295 reap_dead start 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.884+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 0x7fdfa4003e20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009fc0 tx=0x7fdfa4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe68a7ab981152d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.884+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4009c10 con 0x7fdfa40039b0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.884+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdfb4036070 con 0x7fdfa40039b0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.884+0000 7fdfbd1ed640 1 -- 172.21.15.150:0/2884051295 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40318a0 con 0x7fdfa40039b0 2024-04-25T14:07:25.886 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 -- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 msgr2=0x7fdfa4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 0x7fdfa4003e20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009fc0 tx=0x7fdfa4005830 comp rx=0 tx=0).stop 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 -- 172.21.15.150:0/2884051295 shutdown_connections 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 --2- 172.21.15.150:0/2884051295 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfa40039b0 0x7fdfa4003e20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 -- 172.21.15.150:0/2884051295 >> 172.21.15.150:0/2884051295 conn(0x7fdfb80d1ae0 msgr2=0x7fdfb80d1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 -- 172.21.15.150:0/2884051295 shutdown_connections 2024-04-25T14:07:25.887 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.885+0000 7fdfc0453640 1 -- 172.21.15.150:0/2884051295 wait complete. 2024-04-25T14:07:25.889 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.886+0000 7fdfc0453640 1 Processor -- start 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.886+0000 7fdfc0453640 1 -- start start 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 --2- >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa40039b0 0x7fdfb81e7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 --2- >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8142830 0x7fdfb8142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 -- --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80ae960 con 0x7fdfb81453f0 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 -- --> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb80ae7e0 con 0x7fdfb8142830 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 -- --> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb8161f20 con 0x7fdfa40039b0 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 --2- >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.150:3300/0 says I am v2:172.21.15.150:53472/0 (socket says 172.21.15.150:53472) 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 -- 172.21.15.150:0/3690798240 learned_addr learned my addr 172.21.15.150:0/3690798240 (peer_addr_for_me v2:172.21.15.150:0/0) 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbd9ee640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8142830 0x7fdfb8142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa40039b0 0x7fdfb81e7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 -- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa40039b0 msgr2=0x7fdfb81e7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.890 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa40039b0 0x7fdfb81e7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 -- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8142830 msgr2=0x7fdfb8142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8142830 0x7fdfb8142c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 -- 172.21.15.150:0/3690798240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb8145d40 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfbe9f0640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806e080 tx=0x7fdfb000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0019070 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fdfb0007e30 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb00035f0 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.888+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb80b5510 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.889+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb80b5ad0 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.889+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb80b5280 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.890+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 4 ==== mgrmap(e 5) ==== 96782+0+0 (secure 0 0 0) 0x7fdfb001c070 con 0x7fdfb81453f0 2024-04-25T14:07:25.891 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.890+0000 7fdfaeffd640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 0x7fdf9c0769c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T14:07:25.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.891+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 5 ==== osd_map(65..65 src has 1..65) ==== 5421+0+0 (secure 0 0 0) 0x7fdfb0094540 con 0x7fdfb81453f0 2024-04-25T14:07:25.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.891+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 0x7fdf9c0769c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T14:07:25.892 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.891+0000 7fdfbe1ef640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 0x7fdf9c0769c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4035000 tx=0x7fdfb402f000 comp rx=0 tx=0).ready entity=mgr.4110 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T14:07:25.894 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:25.895+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfb8145d40 con 0x7fdfb81453f0 2024-04-25T14:07:26.104 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.105+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 --> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfb8076a40 con 0x7fdfb81453f0 2024-04-25T14:07:26.105 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.106+0000 7fdfaeffd640 1 -- 172.21.15.150:0/3690798240 <== mon.0 v2:172.21.15.150:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) ==== 94+0+3878 (secure 0 0 0) 0x7fdfb0065e20 con 0x7fdfb81453f0 2024-04-25T14:07:26.105 INFO:teuthology.orchestra.run.smithi150.stdout: 2024-04-25T14:07:26.105 INFO:teuthology.orchestra.run.smithi150.stdout:{"epoch":39,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5306,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6834/734824450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6832","nonce":734824450},{"type":"v1","addr":"172.21.15.187:6834","nonce":734824450}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6837/3051398501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6835","nonce":3051398501},{"type":"v1","addr":"172.21.15.150:6837","nonce":3051398501}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":32},{"gid":5728,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.187:6835/3707723519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.187:6833","nonce":3707723519},{"type":"v1","addr":"172.21.15.187:6835","nonce":3707723519}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":33},{"gid":6195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.150:6836/1099437657","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.150:6834","nonce":1099437657},{"type":"v1","addr":"172.21.15.150:6836","nonce":1099437657}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":39}],"filesystems":[]} 2024-04-25T14:07:26.105 INFO:teuthology.orchestra.run.smithi150.stderr:dumped fsmap epoch 39 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 msgr2=0x7fdf9c0769c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 0x7fdf9c0769c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4035000 tx=0x7fdfb402f000 comp rx=0 tx=0).stop 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 msgr2=0x7fdfb8145800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fdfb806e080 tx=0x7fdfb000e820 comp rx=0 tx=0).stop 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 shutdown_connections 2024-04-25T14:07:26.107 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:6832/3849822706,v1:172.21.15.150:6833/3849822706] conn(0x7fdf9c0744b0 0x7fdf9c0769c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7fdfb81453f0 0x7fdfb8145800 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3300/0,v1:172.21.15.187:6789/0] conn(0x7fdfb8142830 0x7fdfb8142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 --2- 172.21.15.150:0/3690798240 >> [v2:172.21.15.187:3301/0,v1:172.21.15.187:6790/0] conn(0x7fdfa40039b0 0x7fdfb81e7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 >> 172.21.15.150:0/3690798240 conn(0x7fdfb80d1ae0 msgr2=0x7fdfb8068690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 shutdown_connections 2024-04-25T14:07:26.108 INFO:teuthology.orchestra.run.smithi150.stderr:2024-04-25T14:07:26.108+0000 7fdfc0453640 1 -- 172.21.15.150:0/3690798240 wait complete. 2024-04-25T14:07:26.118 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-25T14:07:26.118 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-25T14:07:26.119 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-25T14:07:26.119 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.130 INFO:tasks.ceph.mds.a:Stopped 2024-04-25T14:07:26.130 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-25T14:07:26.130 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.179 INFO:tasks.ceph.mds.c:Stopped 2024-04-25T14:07:26.180 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-25T14:07:26.180 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.192 INFO:tasks.ceph.mds.b:Stopped 2024-04-25T14:07:26.192 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-25T14:07:26.192 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.244 INFO:tasks.ceph.mds.d:Stopped 2024-04-25T14:07:26.244 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-25T14:07:26.244 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-25T14:07:26.244 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.269 INFO:tasks.ceph.osd.0:Stopped 2024-04-25T14:07:26.269 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-25T14:07:26.269 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.340 INFO:tasks.ceph.osd.1:Stopped 2024-04-25T14:07:26.340 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-25T14:07:26.340 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.409 INFO:tasks.ceph.osd.2:Stopped 2024-04-25T14:07:26.409 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-25T14:07:26.409 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.475 INFO:tasks.ceph.osd.3:Stopped 2024-04-25T14:07:26.475 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-25T14:07:26.475 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.511 INFO:tasks.ceph.osd.4:Stopped 2024-04-25T14:07:26.512 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-25T14:07:26.512 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.580 INFO:tasks.ceph.osd.5:Stopped 2024-04-25T14:07:26.580 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-25T14:07:26.580 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.647 INFO:tasks.ceph.osd.6:Stopped 2024-04-25T14:07:26.647 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-25T14:07:26.648 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.717 INFO:tasks.ceph.osd.7:Stopped 2024-04-25T14:07:26.717 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-25T14:07:26.717 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-25T14:07:26.717 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.743 INFO:tasks.ceph.mgr.y:Stopped 2024-04-25T14:07:26.743 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-25T14:07:26.743 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.783 INFO:tasks.ceph.mgr.x:Stopped 2024-04-25T14:07:26.784 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-25T14:07:26.784 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-25T14:07:26.784 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.795 INFO:tasks.ceph.mon.a:Stopped 2024-04-25T14:07:26.795 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-25T14:07:26.795 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.838 INFO:tasks.ceph.mon.b:Stopped 2024-04-25T14:07:26.838 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-25T14:07:26.838 INFO:teuthology.orchestra.run:waiting for 300 2024-04-25T14:07:26.891 INFO:tasks.ceph.mon.c:Stopped 2024-04-25T14:07:26.891 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-25T14:07:26.891 DEBUG:teuthology.orchestra.run.smithi150:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-25T14:07:26.921 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:07:26.921 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-25T14:07:27.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:07:27.143 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-25T14:07:27.234 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:07:27.235 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-25T14:07:27.292 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:07:27.292 DEBUG:teuthology.orchestra.run.smithi150:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-25T14:07:27.358 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:07:27.358 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-25T14:07:27.474 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:07:27.474 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-25T14:07:27.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:07:27.562 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-25T14:07:27.610 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:07:27.610 DEBUG:teuthology.orchestra.run.smithi187:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-25T14:07:27.659 INFO:tasks.ceph:Archiving mon data... 2024-04-25T14:07:27.660 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/data/mon.a.tgz 2024-04-25T14:07:27.661 DEBUG:teuthology.orchestra.run.smithi150:> mktemp 2024-04-25T14:07:27.675 INFO:teuthology.orchestra.run.smithi150.stdout:/tmp/tmp.o7Za0YoDNY 2024-04-25T14:07:27.676 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.o7Za0YoDNY 2024-04-25T14:07:27.826 DEBUG:teuthology.orchestra.run.smithi150:> sudo chmod 0666 /tmp/tmp.o7Za0YoDNY 2024-04-25T14:07:27.907 DEBUG:teuthology.orchestra.remote:smithi150:/tmp/tmp.o7Za0YoDNY is 387KB 2024-04-25T14:07:27.975 DEBUG:teuthology.orchestra.run.smithi150:> rm -fr /tmp/tmp.o7Za0YoDNY 2024-04-25T14:07:27.990 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/data/mon.b.tgz 2024-04-25T14:07:27.990 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-25T14:07:28.007 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.8DtrzsDRIc 2024-04-25T14:07:28.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.8DtrzsDRIc 2024-04-25T14:07:28.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.8DtrzsDRIc 2024-04-25T14:07:28.270 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.8DtrzsDRIc is 512KB 2024-04-25T14:07:28.339 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.8DtrzsDRIc 2024-04-25T14:07:28.353 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/data/mon.c.tgz 2024-04-25T14:07:28.353 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2024-04-25T14:07:28.406 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.l9tKEO7LZn 2024-04-25T14:07:28.407 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.l9tKEO7LZn 2024-04-25T14:07:28.568 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.l9tKEO7LZn 2024-04-25T14:07:28.649 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.l9tKEO7LZn is 511KB 2024-04-25T14:07:28.746 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.l9tKEO7LZn 2024-04-25T14:07:28.760 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-25T14:07:28.760 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T14:07:28.763 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-25T14:07:28.843 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T14:07:28.878 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-25T14:07:28.878 INFO:tasks.ceph:Archiving crash dumps... 2024-04-25T14:07:28.879 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi150/crash 2024-04-25T14:07:28.880 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T14:07:28.907 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi187/crash 2024-04-25T14:07:28.908 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-25T14:07:28.941 INFO:tasks.ceph:Shutting down logrotate 2024-04-25T14:07:28.941 DEBUG:teuthology.orchestra.run.smithi150:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:28.952 DEBUG:teuthology.orchestra.run.smithi187:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:29.025 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:29.048 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-25T14:07:29.087 INFO:tasks.ceph:Compressing logs... 2024-04-25T14:07:29.088 DEBUG:teuthology.orchestra.run.smithi150:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T14:07:29.091 DEBUG:teuthology.orchestra.run.smithi187:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T14:07:29.112 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42954.log 2024-04-25T14:07:29.112 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42834.log 2024-04-25T14:07:29.112 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47148.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34780.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34489.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph.audit.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.34780.log.gz -5 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr: --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34489.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.7.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34489.log.gz 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41066.log 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr: 92.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T14:07:29.174 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.3.41066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.41066.log.gz 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34613.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34569.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr: 93.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34613.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34569.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34613.log.gz 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34569.log.gz 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34700.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34740.log 2024-04-25T14:07:29.175 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34700.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34700.log.gz 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34529.log 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34740.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34740.log.gz 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34660.log 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34529.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34529.log.gz 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.admin.34660.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34660.log.gz 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-25T14:07:29.176 INFO:teuthology.orchestra.run.smithi187.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-25T14:07:29.184 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.40669.log 2024-04-25T14:07:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-mon.b.log: 92.1% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-25T14:07:29.200 INFO:teuthology.orchestra.run.smithi187.stderr:/var/log/ceph/ceph-client.1.40669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.40669.log.gz 2024-04-25T14:07:29.253 INFO:teuthology.orchestra.run.smithi187.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46644.log 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42954.log.gz 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47148.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42834.log.gz 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47553.log 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47148.log.gz 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43507.log 2024-04-25T14:07:29.277 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46644.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45030.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43511.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47553.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43507.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53416.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48657.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45030.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53416.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43511.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53157.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53416.log.gz 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49755.log 2024-04-25T14:07:29.278 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56953.log 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48657.log.gz 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49755.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53157.log.gz 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48909.log 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49755.log.gz 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41688.log 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46854.log 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56953.log.gz 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41688.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48909.log.gz 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44820.log 2024-04-25T14:07:29.279 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46854.log.gz 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44820.log: 91.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.41688.log.gz 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45744.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44820.log.gz/var/log/ceph/ceph-client.admin.42524.log: 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48867.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45744.log: gzip -5 88.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.44183.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45744.log.gz 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34792.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44183.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.48867.log.gz -5 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46728.log 2024-04-25T14:07:29.280 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54878.log 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34792.log.gz 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47106.log 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47022.log 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46728.log.gz 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47106.log.gz 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47022.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.54878.log.gz 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52237.log 2024-04-25T14:07:29.281 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49671.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47022.log.gz 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50234.log 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48345.log 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42398.log 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52237.log.gz 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50234.log.gz 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5/var/log/ceph/ceph-client.admin.48345.log: --verbose -- /var/log/ceph/ceph-client.admin.45123.log 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42398.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-25T14:07:29.282 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48345.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45123.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42398.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48099.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49713.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1%/var/log/ceph/ceph-client.admin.48099.log: -- replaced with /var/log/ceph/ceph-client.admin.45123.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43512.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55399.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48099.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43512.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49713.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49337.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 86.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43512.log.gz 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45609.log 2024-04-25T14:07:29.283 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49337.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.55399.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56868.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.45609.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49337.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.45609.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46770.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46434.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50062.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56868.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50150.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46770.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50062.log: gzip 88.7% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.46434.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.53336.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50150.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.50062.log.gz 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48429.log 2024-04-25T14:07:29.284 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50150.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53336.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose --/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.47637.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47637.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48429.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47064.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40907.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47637.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47232.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52751.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47232.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47064.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56299.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47232.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53115.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52751.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50192.log 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53115.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56299.log.gz 2024-04-25T14:07:29.285 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47469.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50192.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53115.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52961.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50192.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49460.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47469.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46182.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52961.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55902.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53031.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55902.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49460.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48951.log 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53031.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55902.log.gz 2024-04-25T14:07:29.286 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57499.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53031.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55337.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57499.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48951.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42750.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55337.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57499.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50570.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55337.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55514.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52026.log 89.0% 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42750.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44424.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52026.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55514.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40992.log 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.50570.log.gz 2024-04-25T14:07:29.287 INFO:teuthology.orchestra.run.smithi150.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.52026.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53073.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44424.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40992.log: /var/log/ceph/ceph-client.admin.53073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56112.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40992.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53073.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53624.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40730.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53624.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56112.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41520.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53624.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57373.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40730.log.gz 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43248.log 2024-04-25T14:07:29.288 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50318.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57373.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41520.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57289.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57373.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43248.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55089.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50318.log: /var/log/ceph/ceph-client.admin.57289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48555.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50318.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57289.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54615.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55089.log: /var/log/ceph/ceph-client.admin.48555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48141.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55089.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54615.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48555.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43038.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54615.log.gz 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55275.log 2024-04-25T14:07:29.289 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46518.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48141.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55275.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43038.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55275.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46350.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46518.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52583.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46812.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46350.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34424.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52583.log.gz 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46812.log: -- /var/log/ceph/ceph-client.admin.50444.log 2024-04-25T14:07:29.290 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51195.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46812.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50444.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.34424.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50444.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56995.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50276.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56995.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50969.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51195.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50276.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56995.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43835.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50276.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46938.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53540.log 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50969.log.gz 2024-04-25T14:07:29.291 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46938.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43835.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53838.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.46938.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53540.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53838.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53540.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56826.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.3.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53838.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55213.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42996.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55213.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56826.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55640.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55213.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48513.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42996.log: /var/log/ceph/ceph-client.admin.55640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54006.log 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42996.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48513.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55640.log.gz 2024-04-25T14:07:29.292 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53582.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48513.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49502.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54006.log: /var/log/ceph/ceph-client.admin.53582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49797.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54006.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49502.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53582.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52279.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46224.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49502.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49797.log: /var/log/ceph/ceph-client.admin.52279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55682.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49797.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52279.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47883.log 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46224.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42482.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55682.log.gz 2024-04-25T14:07:29.293 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44904.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56773.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47883.log: /var/log/ceph/ceph-client.admin.44904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53666.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47883.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%/var/log/ceph/ceph-client.admin.56773.log: -- replaced with /var/log/ceph/ceph-client.admin.44904.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56911.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56773.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41118.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53666.log: /var/log/ceph/ceph-client.admin.56911.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53666.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41350.log 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56911.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.c.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz 2024-04-25T14:07:29.294 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54836.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49927.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41350.log: /var/log/ceph/ceph-client.admin.54836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41350.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54836.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44180.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49978.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41308.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49927.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46308.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44180.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49978.log: /var/log/ceph/ceph-client.admin.46308.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.1.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49978.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr:gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46308.log.gz 2024-04-25T14:07:29.295 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44046.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53922.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44046.log.gz 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46980.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40949.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53922.log.gz 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50402.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57457.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46980.log: /var/log/ceph/ceph-client.admin.50402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34689.log 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57457.log: 88.7% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46980.log.gz 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50402.log.gz 2024-04-25T14:07:29.296 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57457.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44382.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.34689.log: /var/log/ceph/ceph-client.admin.44382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46686.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-osd.0.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34689.log.gz 88.8% 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44382.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45072.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43514.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46686.log: /var/log/ceph/ceph-client.admin.45072.log: 94.0% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46686.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45072.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56412.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43506.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56412.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43514.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40826.log 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56412.log.gz 2024-04-25T14:07:29.297 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56454.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43506.log.gz 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56454.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40826.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47316.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56454.log.gz 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43847.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44694.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7%/var/log/ceph/ceph-client.admin.43847.log: -- replaced with /var/log/ceph/ceph-client.admin.47316.log.gz 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43847.log.gz 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.42912.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41604.log 2024-04-25T14:07:29.298 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44694.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53710.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42912.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52457.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41604.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57163.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52457.log: /var/log/ceph/ceph-client.admin.53710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57163.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52457.log.gz 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53710.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57163.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41436.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5/var/log/ceph/ceph-client.admin.44181.log: --verbose -- /var/log/ceph/ceph-client.admin.48183.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52625.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41436.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50528.log 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48183.log.gz 2024-04-25T14:07:29.299 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42624.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52625.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50528.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42624.log: /var/log/ceph/ceph-client.admin.51093.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42624.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50020.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41478.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48699.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1%/var/log/ceph/ceph-mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.50020.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49883.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54048.log 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49883.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48699.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49883.log.gz 2024-04-25T14:07:29.300 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55151.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.54470.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55151.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54048.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55151.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57415.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.0.54470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47679.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.54470.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57121.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57415.log.gz/var/log/ceph/ceph-client.admin.47679.log: 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53880.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47679.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56731.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53880.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57121.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49035.log 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53880.log.gz 2024-04-25T14:07:29.301 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56731.log.gz/var/log/ceph/ceph-client.admin.49035.log: 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50108.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49035.log.gz 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56341.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50108.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gz 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50108.log.gz 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.56341.log.gz 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51153.log 2024-04-25T14:07:29.302 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.50824.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.51153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52835.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9%/var/log/ceph/ceph-client.0.50824.log: -- replaced with /var/log/ceph/ceph-client.admin.51153.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.50824.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43850.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43513.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43850.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52835.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43850.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43849.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41204.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43513.log.gz 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55860.log 2024-04-25T14:07:29.303 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41394.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43849.log.gz 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42666.log 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41034.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55860.log.gz 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48267.log 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57599.log 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48267.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42666.log.gz 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48267.log.gz 2024-04-25T14:07:29.304 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54695.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53199.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54695.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57599.log.gz 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41562.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.54695.log.gz 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53199.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53199.log.gz 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47841.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3%/var/log/ceph/ceph-client.admin.47841.log: -- replaced with /var/log/ceph/ceph-client.admin.41562.log.gz 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47841.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41160.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41076.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50636.log 2024-04-25T14:07:29.305 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56647.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41160.log.gz 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42440.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50636.log.gz 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49172.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.56647.log.gz 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48825.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49172.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42440.log.gz 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44186.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49172.log.gz 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53498.log 2024-04-25T14:07:29.306 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44186.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48825.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44186.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41754.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43844.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.53498.log: /var/log/ceph/ceph-client.admin.41754.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53498.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41754.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43332.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46266.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43332.log: /var/log/ceph/ceph-client.admin.43844.log: 89.0% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43844.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43332.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43080.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53754.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46266.log.gz 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52499.log 2024-04-25T14:07:29.307 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1%/var/log/ceph/ceph-client.admin.53754.log: -- replaced with /var/log/ceph/ceph-client.admin.43080.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53754.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44592.log 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44508.log 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52499.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44592.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28406.log 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44508.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44592.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44508.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57641.log 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph.tmp-client.admin.28406.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47967.log 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.28406.log.gz 2024-04-25T14:07:29.308 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44182.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47967.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57641.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47967.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43164.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47925.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44182.log: /var/log/ceph/ceph-client.admin.43164.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44182.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43164.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53241.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57559.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47925.log: /var/log/ceph/ceph-client.admin.53241.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47925.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 89.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42566.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53241.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43509.log 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42566.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57559.log.gz 2024-04-25T14:07:29.309 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42566.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46048.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47595.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43509.log: /var/log/ceph/ceph-client.2.46048.log: 0.0% 87.3% -- replaced with /var/log/ceph/ceph-client.admin.43509.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.2.46048.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52709.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46392.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47595.log: /var/log/ceph/ceph-client.admin.52709.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47595.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52709.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52415.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48471.log 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46392.log: /var/log/ceph/ceph-client.admin.52415.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46392.log.gz 2024-04-25T14:07:29.310 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52415.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56238.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57331.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48471.log: /var/log/ceph/ceph-client.admin.56238.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48471.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56238.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43840.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56070.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43840.log: /var/log/ceph/ceph-client.admin.57331.log: 88.3% 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57331.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43840.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54282.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56070.log: /var/log/ceph/ceph-client.admin.54174.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56070.log.gz 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48057.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44736.log 2024-04-25T14:07:29.311 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54282.log: /var/log/ceph/ceph-client.admin.48057.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48057.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54282.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56196.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43845.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44736.log: /var/log/ceph/ceph-client.admin.56196.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44736.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.56196.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.53964.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52153.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1%/var/log/ceph/ceph-client.admin.53964.log: -- replaced with /var/log/ceph/ceph-client.admin.43845.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44550.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53964.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45290.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52153.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44550.log.gz 2024-04-25T14:07:29.312 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.45290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47799.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44178.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45290.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54216.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47511.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54216.log: /var/log/ceph/ceph-client.admin.47799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47274.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47799.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49629.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 89.1%/var/log/ceph/ceph-client.admin.47274.log: -- replaced with /var/log/ceph/ceph-client.admin.47511.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.54216.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44652.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42708.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44652.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49629.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52919.log 2024-04-25T14:07:29.313 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44652.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42792.log 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42708.log: /var/log/ceph/ceph-client.admin.52919.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52541.log 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42792.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42708.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52919.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43206.log 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42792.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46602.log 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52541.log: /var/log/ceph/ceph-client.admin.43206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44988.log 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46602.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43206.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46602.log.gz 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52541.log.gz/var/log/ceph/ceph-client.admin.44988.log: 2024-04-25T14:07:29.314 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52667.log 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44988.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip -5 --verbose -- 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: /var/log/ceph/ceph-client.admin.54944.log 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49839.log 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52667.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49120.log 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54944.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56154.log 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49839.log.gz/var/log/ceph/ceph-client.admin.49120.log: 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43841.log 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49120.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56154.log: gzip -5 --verbose 87.5% -- replaced with /var/log/ceph/ceph-client.admin.56154.log.gz 2024-04-25T14:07:29.315 INFO:teuthology.orchestra.run.smithi150.stderr: -- /var/log/ceph/ceph-client.admin.48741.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55986.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43841.log.gz/var/log/ceph/ceph-client.admin.48741.log: 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48225.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48741.log.gz 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48993.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55986.log: /var/log/ceph/ceph-client.admin.48225.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48993.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55986.log.gz 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48225.log.gz 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44862.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48993.log.gz 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48387.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.44862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44466.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.48387.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44862.log.gz 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48387.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49418.log 2024-04-25T14:07:29.316 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44466.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44466.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52195.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57205.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49418.log: /var/log/ceph/ceph-client.admin.52195.log: 88.8% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49418.log.gz -- replaced with /var/log/ceph/ceph-client.admin.52195.log.gz 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56689.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55776.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57205.log.gz 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.56689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49257.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55776.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56689.log.gz 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55776.log.gzgzip 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41646.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55944.log 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.41646.log: /var/log/ceph/ceph-client.admin.49257.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49257.log.gz 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41646.log.gz 2024-04-25T14:07:29.317 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47358.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52321.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55944.log: /var/log/ceph/ceph-client.admin.47358.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55944.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47358.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49545.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47190.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.52321.log: /var/log/ceph/ceph-client.admin.49545.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52321.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49545.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43122.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43510.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43122.log: /var/log/ceph/ceph-client.admin.47190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47190.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43122.log.gz 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56028.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57247.log 2024-04-25T14:07:29.318 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43510.log: /var/log/ceph/ceph-client.admin.56028.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43510.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56028.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55818.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46896.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55818.log: /var/log/ceph/ceph-client.admin.57247.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55818.log.gz 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57247.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41248.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50486.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46896.log: /var/log/ceph/ceph-client.admin.41248.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46896.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41248.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48783.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42356.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50486.log: /var/log/ceph/ceph-client.admin.48783.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50486.log.gz 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48783.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54090.log 2024-04-25T14:07:29.319 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54132.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.42356.log: /var/log/ceph/ceph-client.admin.54090.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54090.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42356.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50360.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.54132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44946.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54132.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.50360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43290.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.44946.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50360.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44946.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52877.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49215.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.43290.log: /var/log/ceph/ceph-client.admin.52877.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52877.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43290.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55556.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47757.log 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.49215.log.gz 2024-04-25T14:07:29.320 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48597.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.47757.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55556.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47757.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49077.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46560.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.49077.log: /var/log/ceph/ceph-client.admin.48597.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48597.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49077.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46476.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46560.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55598.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46560.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.46476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57037.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.55598.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46476.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55598.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44179.log 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr:/var/log/ceph/ceph-client.admin.57037.log: /var/log/ceph/ceph-client.admin.44179.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57037.log.gz 2024-04-25T14:07:29.321 INFO:teuthology.orchestra.run.smithi150.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44179.log.gz 2024-04-25T14:07:29.322 INFO:teuthology.orchestra.run.smithi150.stderr: 91.9% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-25T14:07:29.322 INFO:teuthology.orchestra.run.smithi187.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-25T14:07:29.335 INFO:teuthology.orchestra.run.smithi187.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-25T14:07:29.602 INFO:teuthology.orchestra.run.smithi150.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-25T14:07:29.879 INFO:teuthology.orchestra.run.smithi150.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-25T14:07:29.881 INFO:teuthology.orchestra.run.smithi187.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-25T14:07:29.945 INFO:teuthology.orchestra.run.smithi187.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-25T14:07:29.970 INFO:teuthology.orchestra.run.smithi150.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-25T14:07:29.971 INFO:teuthology.orchestra.run.smithi150.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-25T14:07:29.973 INFO:teuthology.orchestra.run.smithi187.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-25T14:07:29.975 INFO:teuthology.orchestra.run.smithi187.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-25T14:07:29.977 INFO:teuthology.orchestra.run.smithi187.stderr: 2024-04-25T14:07:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:real 0m0.832s 2024-04-25T14:07:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:user 0m3.459s 2024-04-25T14:07:29.977 INFO:teuthology.orchestra.run.smithi187.stderr:sys 0m0.142s 2024-04-25T14:07:30.020 INFO:teuthology.orchestra.run.smithi150.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-25T14:07:30.022 INFO:teuthology.orchestra.run.smithi150.stderr: 2024-04-25T14:07:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:real 0m0.919s 2024-04-25T14:07:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:user 0m3.825s 2024-04-25T14:07:30.022 INFO:teuthology.orchestra.run.smithi150.stderr:sys 0m0.323s 2024-04-25T14:07:30.023 INFO:tasks.ceph:Archiving logs... 2024-04-25T14:07:30.023 DEBUG:teuthology.misc:Transferring archived files from smithi150:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi150/log 2024-04-25T14:07:30.024 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T14:07:30.848 DEBUG:teuthology.misc:Transferring archived files from smithi187:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi187/log 2024-04-25T14:07:30.849 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T14:07:31.688 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T14:07:31.698 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T14:07:31.699 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T14:07:31.702 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T14:07:31.789 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T14:07:31.799 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T14:07:31.799 DEBUG:teuthology.orchestra.run.smithi150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T14:07:31.802 DEBUG:teuthology.orchestra.run.smithi187:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T14:07:31.816 INFO:teuthology.orchestra.run.smithi150.stderr:bash: line 1: ntpq: command not found 2024-04-25T14:07:31.819 INFO:teuthology.orchestra.run.smithi187.stderr:bash: line 1: ntpq: command not found 2024-04-25T14:07:31.820 INFO:teuthology.orchestra.run.smithi150.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T14:07:31.821 INFO:teuthology.orchestra.run.smithi150.stdout:=============================================================================== 2024-04-25T14:07:31.821 INFO:teuthology.orchestra.run.smithi150.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +1147us[+1155us] +/- 63ms 2024-04-25T14:07:31.821 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 31 -1542us[-1534us] +/- 54ms 2024-04-25T14:07:31.821 INFO:teuthology.orchestra.run.smithi150.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +714us[ +722us] +/- 79ms 2024-04-25T14:07:31.821 INFO:teuthology.orchestra.run.smithi150.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:=============================================================================== 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 +1176us[+1190us] +/- 63ms 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -1496us[-1496us] +/- 54ms 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +745us[ +759us] +/- 79ms 2024-04-25T14:07:31.824 INFO:teuthology.orchestra.run.smithi187.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T14:07:31.825 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T14:07:31.853 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T14:07:31.854 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T14:07:31.964 DEBUG:teuthology.orchestra.run.smithi150:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T14:07:32.037 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T14:07:32.104 DEBUG:teuthology.orchestra.run.smithi150:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T14:07:32.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:07:32.134 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T14:07:32.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:07:32.162 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T14:07:32.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T14:07:32.216 INFO:teuthology.task.internal:Duration was 700.164849 seconds 2024-04-25T14:07:32.217 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T14:07:32.266 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T14:07:32.266 DEBUG:teuthology.orchestra.run.smithi150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T14:07:32.269 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T14:07:32.305 INFO:teuthology.orchestra.run.smithi150.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T14:07:32.309 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T14:07:32.797 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T14:07:32.797 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi150.front.sepia.ceph.com 2024-04-25T14:07:32.798 DEBUG:teuthology.orchestra.run.smithi150:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T14:07:32.859 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2024-04-25T14:07:32.859 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T14:07:32.899 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T14:07:32.899 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T14:07:32.902 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T14:07:32.966 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T14:07:32.967 DEBUG:teuthology.orchestra.run.smithi150:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T14:07:32.970 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T14:07:33.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T14:07:33.115 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T14:07:33.116 DEBUG:teuthology.orchestra.run.smithi150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T14:07:33.139 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T14:07:33.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T14:07:33.174 DEBUG:teuthology.orchestra.run.smithi150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:07:33.182 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:07:33.202 INFO:teuthology.orchestra.run.smithi150.stdout:kernel.core_pattern = core 2024-04-25T14:07:33.224 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2024-04-25T14:07:33.237 DEBUG:teuthology.orchestra.run.smithi150:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:07:33.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:07:33.269 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T14:07:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T14:07:33.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T14:07:33.300 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T14:07:33.301 DEBUG:teuthology.misc:Transferring archived files from smithi150:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi150 2024-04-25T14:07:33.301 DEBUG:teuthology.orchestra.run.smithi150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T14:07:33.342 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673193/remote/smithi187 2024-04-25T14:07:33.342 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T14:07:33.375 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T14:07:33.375 DEBUG:teuthology.orchestra.run.smithi150:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T14:07:33.379 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T14:07:33.424 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T14:07:33.435 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T14:07:33.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T14:07:33.502 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T14:07:33.502 DEBUG:teuthology.orchestra.run.smithi150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T14:07:33.504 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T14:07:33.516 INFO:teuthology.orchestra.run.smithi150.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 14:07 /home/ubuntu/cephtest 2024-04-25T14:07:33.518 INFO:teuthology.orchestra.run.smithi187.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 14:07 /home/ubuntu/cephtest 2024-04-25T14:07:33.519 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T14:07:33.558 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T14:07:33.645 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/openfiletable} duration: 700.1648490428925 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T14:07:33.645 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T14:07:33.724 INFO:teuthology.run:pass